Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X........................................X..] duration: 5.784054513s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:31:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.784083779s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 816.605059ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:31:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 816.63719ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........XXXXXXXXX.....................................X..] duration: 3.484395542s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:30:52 UTC) orbiter warning: context deadline exceeded Duration: 3.484425819s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 794.850744ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:30:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 794.871994ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [............XXXXXXXXX.....................................X.] duration: 3.464979009s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:30:33 UTC) orbiter warning: context deadline exceeded Duration: 3.465009338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X........................................X.] duration: 6.304247239s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:30:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.304273679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X........................................X] duration: 30.260925495s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:30:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.260994828s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 980.897002ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:30:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 980.929825ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............XXXXXXXXX.....................................X] duration: 5.00158569s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:29:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001609655s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 810.356498ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:29:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 810.475837ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.232152778s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:29:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.232185971s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 830.967778ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:28:44 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 830.997214ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..............XXXXXXXXX.....................................] duration: 3.4914425s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:28:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491467417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.931513026s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:28:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.931549765s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 800.196363ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:28:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 800.249585ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...............XXXXXXXXX....................................] duration: 3.542167586s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:27:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542197223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.970871752s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:27:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.970906809s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 842.448077ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:27:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 842.476522ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.009496089s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:26:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.009517329s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [................XXXXXXXXX...................................] duration: 3.457247613s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:26:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457280676s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 475.648726ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:25:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 475.668875ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.................XXXXXXXXX..................................] duration: 3.460908445s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:25:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460934905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.77332991s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:25:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.77335592s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..................XXXXXXXXX.................................] duration: 3.511582156s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:25:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511606623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.916190217s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:24:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.916233029s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...................XXXXXXXXX................................] duration: 3.55185211s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:24:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551879432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.911764869s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:24:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.911786702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.521263353s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:24:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.52129181s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....................XXXXXXXXX...............................] duration: 3.504930744s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:23:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505028592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.472828223s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:23:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.472860815s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....................XXXXXXXXX..............................] duration: 3.537808377s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:23:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537880603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.004936819s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:22:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.004967335s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......................XXXXXXXXX.............................] duration: 3.438080018s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:22:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438117269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.884526269s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:21:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.884555956s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......................XXXXXXXXX............................] duration: 3.450284707s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:20:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450312371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........................XXXXXXXXX...........................] duration: 3.434105629s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:20:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434136369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.949527457s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:20:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.949554147s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.........................XXXXXXXXX..........................] duration: 3.531114715s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:20:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531146876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..........................XXXXXXXXX.........................] duration: 3.426839171s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:19:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426865521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.89552569s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:19:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.895555005s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...........................XXXXXXXXX........................] duration: 3.400881491s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:18:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.400909765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.787687488s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:18:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.787738084s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [............................XXXXXXXXX.......................] duration: 3.417409346s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:18:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417443531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X................................X..........................] duration: 5.80041692s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:18:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.800442558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X................................X.........................] duration: 5.863131051s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:17:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.863160408s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X............................XXXXXXXXX......................] duration: 3.431465885s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:17:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.431525509s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X............................XXXXXXXXX.....................] duration: 3.423374966s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:16:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.423420073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................................X........................] duration: 5.864139584s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.864164751s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X............................XXXXXXXXX....................] duration: 3.373878684s trace: https://jaeger.dev.storj.io/trace/3d52156582e081c7 FAIL (2025-11-13 20:15:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.373908893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X................................X.......................] duration: 5.912392586s trace: https://jaeger.dev.storj.io/trace/11182b9e24315d73 FAIL (2025-11-13 20:15:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 94917]: Uploading piece... Nov 13 19:57:21.264930947 (107h53m19.816401418s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265358120 (107h53m19.816828581s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.265864886 (107h53m19.817335347s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266024250 (107h53m19.817494711s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266239820 (107h53m19.817710291s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266544300 (107h53m19.818014761s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.266903854 (107h53m19.818374315s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267494960 (107h53m19.818965421s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.267877809 (107h53m19.81934827s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:21.268211114 (107h53m19.819681565s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Uploading piece... Nov 13 19:57:23.504751216 (107h53m22.056221677s) [upload=94917]: Next returned nil segment Nov 13 19:57:23.504768319 (107h53m22.05623877s) [upload=94917]: Waiting for error group managing segments... Nov 13 19:57:24.167251386 (107h53m22.718721847s) [noise=true , address=77.90.31.15:4697 , piece_id=ALAGD23STHNXV4YZSOCFZLYCLIHLCD6YJTMO4QBQSWVGZPC7XW5Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.208034415 (107h53m22.759504876s) [noise=true , address=45.91.116.8:4045 , piece_id=A54BEC4N66HF3W22NBCKUJRAC4YGCXP362R7FQI6N32YNS76HBCQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.291652672 (107h53m22.843123143s) [noise=true , address=45.91.116.7:4009 , piece_id=4CMJFXDGPANW4VEE7RYVZ27WW5S5IPEMDIZPQ2S2U2KMPHPRGR4A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:24.845341442 (107h53m23.396811903s) [noise=true , address=77.90.31.12:4590 , piece_id=MK4WX7EFSVXCA7P2JGRWTHPHLM7WAM2IV2IOVBT42TUS7CAOBI6Q, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.470936266 (107h53m24.022406727s) [noise=true , address=77.90.31.18:4807 , piece_id=VYH276EMB3C5CPREOPPMQ7QDXK7GFHK4PJZ5XZESVOKAFETR5AAA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:25.558314452 (107h53m24.109784903s) [noise=true , address=45.91.116.12:4226 , piece_id=QDD22BSV4TYFAV2BVLM57LYZZELJKDFH3IJJXT6YITHTALNXWNFA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: <nil> Nov 13 19:57:50.037576459 (107h53m48.58904692s) [noise=true , address=145.79.227.27:4691 , piece_id=2OLA627PEAURU7S6XGM2FFVUFN3KBW6KHVJHWOUQZDOTFXLC4YMA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34364->145.79.227.27:4691: use of closed network connection Nov 13 19:57:50.037695305 (107h53m48.589165756s) [noise=true , address=145.79.227.10:4401 , piece_id=2UKYCJGDM4HPJIPZ2V7HC4QNBLKBYDHXQRHZNYFNTTXDZNRIVCXQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36780->145.79.227.10:4401: use of closed network connection Nov 13 19:57:50.037739029 (107h53m48.58920948s) [noise=true , address=145.79.227.8:4329 , piece_id=ALXBLQQFGW4COOH2KOUQMXADDRHYKSJA5YG3CIRU7MFTBPQXUDBQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45540->145.79.227.8:4329: use of closed network connection Nov 13 19:57:50.037785598 (107h53m48.589256049s) [noise=true , address=145.79.227.30:4763 , piece_id=RQNFYEDCM35AKCOEKX6VXQWBPRKZKK2MLYKQX7B4Q7WQXB3XBFVA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59786->145.79.227.30:4763: use of closed network connection Nov 13 19:57:50.037917449 (107h53m48.5893879s) [noise=true , address=145.79.227.32:4834 , piece_id=XO3XWHWAOA47JGRZY6WFFX6YAI2VASMP2HCGAWFCZC7AHE7OUAUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56412->145.79.227.32:4834: use of closed network connection Nov 13 19:57:50.037999755 (107h53m48.589470206s) [noise=true , address=145.79.227.14:4839 , piece_id=WAAE7PU53BRLYV6W5YPCDJTJKOKKE2O5LOIQXOROVQZ3LBB6JFUQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60050->145.79.227.14:4839: use of closed network connection Nov 13 19:57:50.038132218 (107h53m48.589602679s) [noise=true , address=145.79.227.11:4730 , piece_id=SVSX2MT3EC5EQ652WO3FDL4FNG6JEMMNHLZZQCJMNXLSFZTSP5UA, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 19:57:50.038200247 (107h53m48.589670698s) [noise=true , address=145.79.227.26:4654 , piece_id=WGPVV7RJ7XAMDOWW3LDTR5IKOQADXF6BGH45MILY7RU2VH2KR3YQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45074->145.79.227.26:4654: use of closed network connection Nov 13 19:57:50.038350503 (107h53m48.589820964s) [noise=true , address=145.79.227.16:4910 , piece_id=JTZ6ZE6IOIP6FUZINMLLN4ADJYQTBH5C4MOBZVX3SFADBKSI6F7A, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47126->145.79.227.16:4910: use of closed network connection Nov 13 19:57:50.038431116 (107h53m48.589901577s) [noise=true , address=145.79.227.22:4511 , piece_id=JDILBBMHS62CZYCHTKDAPM3ECWRUVJAV3VT543DWY6DUXYJ6M6ZQ, seg_pos={0 0}, upload=94917]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44172->145.79.227.22:4511: use of closed network connection Nov 13 19:57:50.038487444 (107h53m48.589957895s) [seg_pos={0 0}, upload=94917]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 13 19:57:50.362996026 (107h53m48.914466497s) [upload=94917]: Done uploading segments... Nov 13 19:57:50.363037346 (107h53m48.914507797s) [upload=94917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.912423694s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)