Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [XXXX.....................................X..................] duration: 3.475323214s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:42:44 UTC) orbiter warning: context deadline exceeded Duration: 3.475348493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........................................X................] duration: 5.879929074s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:42:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.879964332s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 819.359021ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:42:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 819.423995ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X........................................X...............] duration: 5.844296842s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:41:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.844333934s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [XXXXX.....................................X.................] duration: 3.396356551s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:41:41 UTC) orbiter warning: context deadline exceeded Duration: 3.396383083s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 760.62597ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:41:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 760.64172ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X........................................X..............] duration: 5.905381239s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:41:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.905400546s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 792.098311ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:40:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 792.119371ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X........................................X.............] duration: 5.957132215s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:40:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.957161721s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XXXXXX.....................................X................] duration: 3.427949135s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:40:37 UTC) orbiter warning: context deadline exceeded Duration: 3.427982137s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 744.484937ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:40:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 744.588373ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X........................................X............] duration: 5.979455346s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:39:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.979491485s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XXXXXXX.....................................X...............] duration: 3.422540794s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:39:29 UTC) orbiter warning: context deadline exceeded Duration: 3.422568728s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 798.315832ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:39:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 798.341439ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X........................................X...........] duration: 5.869207356s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:39:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.869242233s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [XXXXXXXX.....................................X..............] duration: 3.451913582s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:39:13 UTC) orbiter warning: context deadline exceeded Duration: 3.451986681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [XXXXXXXXX.....................................X.............] duration: 3.431500376s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:38:39 UTC) orbiter warning: context deadline exceeded Duration: 3.431518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X........................................X..........] duration: 5.959981496s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:38:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.960012856s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 753.58824ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:38:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 753.618648ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.XXXXXXXXX.....................................X............] duration: 3.394804601s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:38:07 UTC) orbiter warning: context deadline exceeded Duration: 3.394847423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X........................................X.........] duration: 5.896753537s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:37:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.89683438s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 749.41751ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:37:53 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 749.446535ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..XXXXXXXXX.....................................X...........] duration: 3.439095329s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:37:20 UTC) orbiter warning: context deadline exceeded Duration: 3.439111901s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 785.12874ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:37:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 785.205826ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...XXXXXXXXX.....................................X..........] duration: 3.409327104s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:36:58 UTC) orbiter warning: context deadline exceeded Duration: 3.409358644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X........................................X........] duration: 6.135499248s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:36:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.135528924s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 869.872109ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:36:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 869.897988ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X........................................X.......] duration: 5.966041884s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:36:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.966075519s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 796.646631ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:35:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 796.677179ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [....XXXXXXXXX.....................................X.........] duration: 3.40985645s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:35:57 UTC) orbiter warning: context deadline exceeded Duration: 3.409884323s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 867.999594ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:35:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 868.025484ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X........................................X......] duration: 6.110838882s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:35:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.110872267s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.....XXXXXXXXX.....................................X........] duration: 3.423330739s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:34:56 UTC) orbiter warning: context deadline exceeded Duration: 3.423362821s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 849.398242ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:34:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 849.425434ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 852.089407ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:34:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 852.127669ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......XXXXXXXXX.....................................X.......] duration: 3.444902748s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:34:10 UTC) orbiter warning: context deadline exceeded Duration: 3.445039378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X........................................X.....] duration: 5.910740802s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:34:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.910777032s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 793.879302ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:33:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 793.908347ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......XXXXXXXXX.....................................X......] duration: 3.494561047s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:33:36 UTC) orbiter warning: context deadline exceeded Duration: 3.494575394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X........................................X....] duration: 6.039152405s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:33:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.039183164s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 815.200954ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:33:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 815.230139ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [........XXXXXXXXX.....................................X.....] duration: 3.399213028s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:32:51 UTC) orbiter warning: context deadline exceeded Duration: 3.399238096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X........................................X...] duration: 5.938758871s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:32:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.938877858s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........XXXXXXXXX.....................................X....] duration: 3.526358236s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:32:33 UTC) orbiter warning: context deadline exceeded Duration: 3.526373014s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 841.569253ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:32:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 841.601644ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........XXXXXXXXX.....................................X...] duration: 3.438122726s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:31:46 UTC) orbiter warning: context deadline exceeded Duration: 3.438148897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X........................................X..] duration: 5.784054513s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:31:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.784083779s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 816.605059ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:31:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 816.63719ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........XXXXXXXXX.....................................X..] duration: 3.484395542s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:30:52 UTC) orbiter warning: context deadline exceeded Duration: 3.484425819s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 794.850744ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:30:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 794.871994ms edge authservice http us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)