Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.459312839s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:06:26 UTC) orbiter warning: context deadline exceeded Duration: 3.459338428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.014683006s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:06:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.014712523s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.402989705s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:05:53 UTC) orbiter warning: context deadline exceeded Duration: 3.403017908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.123714004s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:05:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.123741656s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.526332141s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:05:08 UTC) orbiter warning: context deadline exceeded Duration: 3.52635309s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.531148634s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:04:44 UTC) orbiter warning: context deadline exceeded Duration: 3.53118925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.013783509s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:04:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.013803417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.137863614s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:03:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.137891857s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.445100171s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:03:30 UTC) orbiter warning: context deadline exceeded Duration: 3.445132032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.120165853s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:02:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.120202262s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.464032663s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:02:24 UTC) orbiter warning: context deadline exceeded Duration: 3.464060305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.166089511s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:02:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.16612093s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.436944391s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:01:48 UTC) orbiter warning: context deadline exceeded Duration: 3.436973866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.056156168s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:01:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.056184792s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.544403955s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:01:02 UTC) orbiter warning: context deadline exceeded Duration: 3.544450915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.018760628s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:00:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.01880944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.481668753s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:00:27 UTC) orbiter warning: context deadline exceeded Duration: 3.481697657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.012519732s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:59:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.012543307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.012145559s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:59:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.012180044s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.749746128s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:59:16 UTC) orbiter warning: context deadline exceeded Duration: 3.749776165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.058659791s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:58:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.058686852s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.012263042s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:58:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.012296506s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.564386368s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:58:15 UTC) orbiter warning: context deadline exceeded Duration: 3.564406847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.986935452s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:58:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.986986349s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.602224878s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:58:00 UTC) orbiter warning: context deadline exceeded Duration: 3.602254785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.031452142s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:57:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.031480276s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.508392797s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:56:54 UTC) orbiter warning: context deadline exceeded Duration: 3.508420168s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.469017703s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:56:36 UTC) orbiter warning: context deadline exceeded Duration: 3.469045505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.019636304s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:56:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.019954751s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.526294079s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:55:51 UTC) orbiter warning: context deadline exceeded Duration: 3.526344025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.574865844s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:55:25 UTC) orbiter warning: context deadline exceeded Duration: 3.574954473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.991971431s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:55:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.99200199s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.656928227s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:54:55 UTC) orbiter warning: context deadline exceeded Duration: 3.656955739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.011963464s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:54:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.011998821s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.538783637s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:54:30 UTC) orbiter warning: context deadline exceeded Duration: 3.538809025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.996236048s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:54:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.996269051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.14869099s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:53:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.148720425s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.382663478s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:53:36 UTC) orbiter warning: context deadline exceeded Duration: 3.38269656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.182575766s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:53:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.182603368s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 4.320362195s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:52:40 UTC) orbiter warning: context deadline exceeded Duration: 4.320379419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.924559496s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:52:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.924590675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.457848585s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:51:40 UTC) orbiter warning: context deadline exceeded Duration: 3.457875566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.628086754s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:51:25 UTC) orbiter warning: context deadline exceeded Duration: 3.628122302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.023694731s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:51:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.023742712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.542062644s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:51:10 UTC) orbiter warning: context deadline exceeded Duration: 3.54209111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.649250687s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:50:54 UTC) orbiter warning: context deadline exceeded Duration: 3.64927864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.427655413s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:50:27 UTC) orbiter warning: context deadline exceeded Duration: 3.427683557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.25741542s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:50:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.257455086s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.637057467s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 02:49:15 UTC) orbiter warning: context deadline exceeded Duration: 3.637081723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.211452211s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:49:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.21148386s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)