Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.952648562s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:56:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.952684229s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 799.378295ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:56:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 799.40189ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.980382123s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:56:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.980412911s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.44441869s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:56:20 UTC) orbiter warning: context deadline exceeded Duration: 3.444446714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.861715582s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:55:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.861735059s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.485707717s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:55:44 UTC) orbiter warning: context deadline exceeded Duration: 3.485738886s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 817.94592ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:55:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 817.983522ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.76608348s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:55:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.766108819s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.426077107s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:54:51 UTC) orbiter warning: context deadline exceeded Duration: 3.426105782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.35632754s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:54:26 UTC) orbiter warning: context deadline exceeded Duration: 3.356343931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.917969892s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:54:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.918006082s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 845.824287ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:54:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 845.865496ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.014032908s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:53:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.014061341s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 818.708803ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:53:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 818.731585ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.444785083s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:53:21 UTC) orbiter warning: context deadline exceeded Duration: 3.44481569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.920267598s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:53:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.920296933s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 814.143189ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:52:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 814.169679ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.422972425s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:52:36 UTC) orbiter warning: context deadline exceeded Duration: 3.422997292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.912786373s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:52:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.912813834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.052014433s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:52:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.052041236s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 799.31835ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:52:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 799.347838ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.37416493s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:51:33 UTC) orbiter warning: context deadline exceeded Duration: 3.374196761s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.014935981s trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:50:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 1.014952752s edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.895900495s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:50:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.895928217s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.375372394s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:50:40 UTC) orbiter warning: context deadline exceeded Duration: 3.375440935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.852095424s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:50:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.852127956s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 766.603734ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:49:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 766.628031ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.432884018s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:49:43 UTC) orbiter warning: context deadline exceeded Duration: 3.432905669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.930346718s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:49:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.930462718s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.597426411s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:48:58 UTC) orbiter warning: context deadline exceeded Duration: 3.597453043s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 821.513673ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:48:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 821.543952ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.010967758s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:48:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.010987777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.39678542s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:48:14 UTC) orbiter warning: context deadline exceeded Duration: 3.396852308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.847920844s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:47:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.847950079s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 846.886909ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:47:44 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 846.959548ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.418765948s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:47:37 UTC) orbiter warning: context deadline exceeded Duration: 3.418796996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.77838411s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:47:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.778407223s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.517736311s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:46:57 UTC) orbiter warning: context deadline exceeded Duration: 3.517758703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.953820267s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:46:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.953874652s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 809.79579ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:46:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 809.819456ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.108284931s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:46:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.10832032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.457948948s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:46:06 UTC) orbiter warning: context deadline exceeded Duration: 3.457979136s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 754.004363ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:45:49 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 754.059307ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.854499001s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:45:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.854538857s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 754.438938ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:45:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 754.479737ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.408448227s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:45:19 UTC) orbiter warning: context deadline exceeded Duration: 3.408478115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.819546559s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:45:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.819576314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........................................X..................] duration: 5.875250013s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.875274459s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.....................................X.....................] duration: 3.534168375s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:44:18 UTC) orbiter warning: context deadline exceeded Duration: 3.534185619s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 797.305387ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:44:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 797.333399ms edge authservice http us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)