Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.......X......XX.....................X.......] duration: 5.738164349s trace: https://jaeger.dev.storj.io/trace/95534c84b640d45 FAIL (2025-11-27 21:20:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 21:15:49.300568539 (445h11m47.852039s) [upload=392027]: Starting upload Nov 27 21:15:49.300648992 (445h11m47.852119453s) [upload=392027]: Done starting upload Nov 27 21:15:49.300845096 (445h11m47.852315557s) [upload=392027]: Uploading segments... Nov 27 21:15:49.300959323 (445h11m47.852429774s) [upload=392027]: Got next segment. Inline: false Nov 27 21:15:49.301019478 (445h11m47.852489939s) [upload=392027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:15:49.301026451 (445h11m47.852496912s) [upload=392027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:15:49.547343071 (445h11m48.098813532s) [seg_pos={0 0}, upload=392027]: Begin upload segment... Nov 27 21:15:49.547402453 (445h11m48.098872914s) [seg_pos={0 0}, upload=392027]: Done begin upload segment. Nov 27 21:15:49.547441377 (445h11m48.098911838s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.547457709 (445h11m48.09892817s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548172770 (445h11m48.099643231s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548203278 (445h11m48.099673749s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548580657 (445h11m48.100051108s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548745050 (445h11m48.100215501s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548991479 (445h11m48.10046194s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549128791 (445h11m48.100599252s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549445283 (445h11m48.100915754s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550022723 (445h11m48.101493174s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550538054 (445h11m48.102008515s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550650238 (445h11m48.102120689s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551095967 (445h11m48.102566418s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551630003 (445h11m48.103100455s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551905909 (445h11m48.10337637s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.552366957 (445h11m48.103837418s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:51.295819242 (445h11m49.847289693s) [upload=392027]: Next returned nil segment Nov 27 21:15:51.295883825 (445h11m49.847354276s) [upload=392027]: Waiting for error group managing segments... Nov 27 21:15:51.607974413 (445h11m50.159444874s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:51.613231052 (445h11m50.164701514s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.226913164 (445h11m52.778383625s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.255879032 (445h11m52.807349503s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.937012666 (445h11m53.488483147s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:55.385905965 (445h11m53.937376426s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:58.996130974 (445h11m57.547601435s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:02.960102785 (445h12m1.511573296s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:04.751312700 (445h12m3.302783161s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:05.874651098 (445h12m4.426121549s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:06.557657779 (445h12m5.10912824s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:18.597005313 (445h12m17.148475774s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:50930->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597177662 (445h12m17.148648123s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:40618->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597200885 (445h12m17.148671336s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:53296->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597348557 (445h12m17.148819018s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:36742->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597425313 (445h12m17.148895774s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: expected piece hash; context deadline exceeded; EOF Nov 27 21:16:18.597465990 (445h12m17.148936451s) [seg_pos={0 0}, upload=392027]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:16:18.842083808 (445h12m17.393554269s) [upload=392027]: Done uploading segments... Nov 27 21:16:18.842126799 (445h12m17.39359726s) [upload=392027]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.73823828s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.792140573s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:20:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.792172102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.......X......XX.....................X......] duration: 5.753696137s trace: https://jaeger.dev.storj.io/trace/95534c84b640d45 FAIL (2025-11-27 21:20:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 21:15:49.300568539 (445h11m47.852039s) [upload=392027]: Starting upload Nov 27 21:15:49.300648992 (445h11m47.852119453s) [upload=392027]: Done starting upload Nov 27 21:15:49.300845096 (445h11m47.852315557s) [upload=392027]: Uploading segments... Nov 27 21:15:49.300959323 (445h11m47.852429774s) [upload=392027]: Got next segment. Inline: false Nov 27 21:15:49.301019478 (445h11m47.852489939s) [upload=392027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:15:49.301026451 (445h11m47.852496912s) [upload=392027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:15:49.547343071 (445h11m48.098813532s) [seg_pos={0 0}, upload=392027]: Begin upload segment... Nov 27 21:15:49.547402453 (445h11m48.098872914s) [seg_pos={0 0}, upload=392027]: Done begin upload segment. Nov 27 21:15:49.547441377 (445h11m48.098911838s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.547457709 (445h11m48.09892817s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548172770 (445h11m48.099643231s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548203278 (445h11m48.099673749s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548580657 (445h11m48.100051108s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548745050 (445h11m48.100215501s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548991479 (445h11m48.10046194s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549128791 (445h11m48.100599252s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549445283 (445h11m48.100915754s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550022723 (445h11m48.101493174s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550538054 (445h11m48.102008515s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550650238 (445h11m48.102120689s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551095967 (445h11m48.102566418s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551630003 (445h11m48.103100455s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551905909 (445h11m48.10337637s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.552366957 (445h11m48.103837418s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:51.295819242 (445h11m49.847289693s) [upload=392027]: Next returned nil segment Nov 27 21:15:51.295883825 (445h11m49.847354276s) [upload=392027]: Waiting for error group managing segments... Nov 27 21:15:51.607974413 (445h11m50.159444874s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:51.613231052 (445h11m50.164701514s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.226913164 (445h11m52.778383625s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.255879032 (445h11m52.807349503s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.937012666 (445h11m53.488483147s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:55.385905965 (445h11m53.937376426s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:58.996130974 (445h11m57.547601435s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:02.960102785 (445h12m1.511573296s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:04.751312700 (445h12m3.302783161s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:05.874651098 (445h12m4.426121549s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:06.557657779 (445h12m5.10912824s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:18.597005313 (445h12m17.148475774s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:50930->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597177662 (445h12m17.148648123s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:40618->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597200885 (445h12m17.148671336s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:53296->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597348557 (445h12m17.148819018s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:36742->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597425313 (445h12m17.148895774s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: expected piece hash; context deadline exceeded; EOF Nov 27 21:16:18.597465990 (445h12m17.148936451s) [seg_pos={0 0}, upload=392027]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:16:18.842083808 (445h12m17.393554269s) [upload=392027]: Done uploading segments... Nov 27 21:16:18.842126799 (445h12m17.39359726s) [upload=392027]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.753717276s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.884181165s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:20:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.884207665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X.......X......XX.....................X.....] duration: 5.858940851s trace: https://jaeger.dev.storj.io/trace/95534c84b640d45 FAIL (2025-11-27 21:20:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 21:15:49.300568539 (445h11m47.852039s) [upload=392027]: Starting upload Nov 27 21:15:49.300648992 (445h11m47.852119453s) [upload=392027]: Done starting upload Nov 27 21:15:49.300845096 (445h11m47.852315557s) [upload=392027]: Uploading segments... Nov 27 21:15:49.300959323 (445h11m47.852429774s) [upload=392027]: Got next segment. Inline: false Nov 27 21:15:49.301019478 (445h11m47.852489939s) [upload=392027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:15:49.301026451 (445h11m47.852496912s) [upload=392027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:15:49.547343071 (445h11m48.098813532s) [seg_pos={0 0}, upload=392027]: Begin upload segment... Nov 27 21:15:49.547402453 (445h11m48.098872914s) [seg_pos={0 0}, upload=392027]: Done begin upload segment. Nov 27 21:15:49.547441377 (445h11m48.098911838s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.547457709 (445h11m48.09892817s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548172770 (445h11m48.099643231s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548203278 (445h11m48.099673749s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548580657 (445h11m48.100051108s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548745050 (445h11m48.100215501s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548991479 (445h11m48.10046194s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549128791 (445h11m48.100599252s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549445283 (445h11m48.100915754s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550022723 (445h11m48.101493174s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550538054 (445h11m48.102008515s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550650238 (445h11m48.102120689s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551095967 (445h11m48.102566418s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551630003 (445h11m48.103100455s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551905909 (445h11m48.10337637s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.552366957 (445h11m48.103837418s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:51.295819242 (445h11m49.847289693s) [upload=392027]: Next returned nil segment Nov 27 21:15:51.295883825 (445h11m49.847354276s) [upload=392027]: Waiting for error group managing segments... Nov 27 21:15:51.607974413 (445h11m50.159444874s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:51.613231052 (445h11m50.164701514s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.226913164 (445h11m52.778383625s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.255879032 (445h11m52.807349503s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.937012666 (445h11m53.488483147s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:55.385905965 (445h11m53.937376426s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:58.996130974 (445h11m57.547601435s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:02.960102785 (445h12m1.511573296s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:04.751312700 (445h12m3.302783161s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:05.874651098 (445h12m4.426121549s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:06.557657779 (445h12m5.10912824s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:18.597005313 (445h12m17.148475774s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:50930->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597177662 (445h12m17.148648123s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:40618->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597200885 (445h12m17.148671336s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:53296->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597348557 (445h12m17.148819018s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:36742->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597425313 (445h12m17.148895774s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: expected piece hash; context deadline exceeded; EOF Nov 27 21:16:18.597465990 (445h12m17.148936451s) [seg_pos={0 0}, upload=392027]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:16:18.842083808 (445h12m17.393554269s) [upload=392027]: Done uploading segments... Nov 27 21:16:18.842126799 (445h12m17.39359726s) [upload=392027]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.858975748s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.89767688s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:19:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.897709922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X.......X......XX.....................X....] duration: 12.660310338s trace: https://jaeger.dev.storj.io/trace/95534c84b640d45 FAIL (2025-11-27 21:19:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 21:15:49.300568539 (445h11m47.852039s) [upload=392027]: Starting upload Nov 27 21:15:49.300648992 (445h11m47.852119453s) [upload=392027]: Done starting upload Nov 27 21:15:49.300845096 (445h11m47.852315557s) [upload=392027]: Uploading segments... Nov 27 21:15:49.300959323 (445h11m47.852429774s) [upload=392027]: Got next segment. Inline: false Nov 27 21:15:49.301019478 (445h11m47.852489939s) [upload=392027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:15:49.301026451 (445h11m47.852496912s) [upload=392027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:15:49.547343071 (445h11m48.098813532s) [seg_pos={0 0}, upload=392027]: Begin upload segment... Nov 27 21:15:49.547402453 (445h11m48.098872914s) [seg_pos={0 0}, upload=392027]: Done begin upload segment. Nov 27 21:15:49.547441377 (445h11m48.098911838s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.547457709 (445h11m48.09892817s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548172770 (445h11m48.099643231s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548203278 (445h11m48.099673749s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548580657 (445h11m48.100051108s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548745050 (445h11m48.100215501s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548991479 (445h11m48.10046194s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549128791 (445h11m48.100599252s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549445283 (445h11m48.100915754s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550022723 (445h11m48.101493174s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550538054 (445h11m48.102008515s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550650238 (445h11m48.102120689s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551095967 (445h11m48.102566418s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551630003 (445h11m48.103100455s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551905909 (445h11m48.10337637s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.552366957 (445h11m48.103837418s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:51.295819242 (445h11m49.847289693s) [upload=392027]: Next returned nil segment Nov 27 21:15:51.295883825 (445h11m49.847354276s) [upload=392027]: Waiting for error group managing segments... Nov 27 21:15:51.607974413 (445h11m50.159444874s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:51.613231052 (445h11m50.164701514s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.226913164 (445h11m52.778383625s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.255879032 (445h11m52.807349503s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.937012666 (445h11m53.488483147s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:55.385905965 (445h11m53.937376426s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:58.996130974 (445h11m57.547601435s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:02.960102785 (445h12m1.511573296s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:04.751312700 (445h12m3.302783161s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:05.874651098 (445h12m4.426121549s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:06.557657779 (445h12m5.10912824s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:18.597005313 (445h12m17.148475774s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:50930->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597177662 (445h12m17.148648123s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:40618->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597200885 (445h12m17.148671336s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:53296->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597348557 (445h12m17.148819018s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:36742->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597425313 (445h12m17.148895774s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: expected piece hash; context deadline exceeded; EOF Nov 27 21:16:18.597465990 (445h12m17.148936451s) [seg_pos={0 0}, upload=392027]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:16:18.842083808 (445h12m17.393554269s) [upload=392027]: Done uploading segments... Nov 27 21:16:18.842126799 (445h12m17.39359726s) [upload=392027]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 12.660360243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X.......X......XX.....................X...] duration: 12.983563241s trace: https://jaeger.dev.storj.io/trace/95534c84b640d45 FAIL (2025-11-27 21:18:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 21:15:49.300568539 (445h11m47.852039s) [upload=392027]: Starting upload Nov 27 21:15:49.300648992 (445h11m47.852119453s) [upload=392027]: Done starting upload Nov 27 21:15:49.300845096 (445h11m47.852315557s) [upload=392027]: Uploading segments... Nov 27 21:15:49.300959323 (445h11m47.852429774s) [upload=392027]: Got next segment. Inline: false Nov 27 21:15:49.301019478 (445h11m47.852489939s) [upload=392027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:15:49.301026451 (445h11m47.852496912s) [upload=392027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:15:49.547343071 (445h11m48.098813532s) [seg_pos={0 0}, upload=392027]: Begin upload segment... Nov 27 21:15:49.547402453 (445h11m48.098872914s) [seg_pos={0 0}, upload=392027]: Done begin upload segment. Nov 27 21:15:49.547441377 (445h11m48.098911838s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.547457709 (445h11m48.09892817s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548172770 (445h11m48.099643231s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548203278 (445h11m48.099673749s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548580657 (445h11m48.100051108s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548745050 (445h11m48.100215501s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548991479 (445h11m48.10046194s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549128791 (445h11m48.100599252s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549445283 (445h11m48.100915754s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550022723 (445h11m48.101493174s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550538054 (445h11m48.102008515s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550650238 (445h11m48.102120689s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551095967 (445h11m48.102566418s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551630003 (445h11m48.103100455s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551905909 (445h11m48.10337637s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.552366957 (445h11m48.103837418s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:51.295819242 (445h11m49.847289693s) [upload=392027]: Next returned nil segment Nov 27 21:15:51.295883825 (445h11m49.847354276s) [upload=392027]: Waiting for error group managing segments... Nov 27 21:15:51.607974413 (445h11m50.159444874s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:51.613231052 (445h11m50.164701514s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.226913164 (445h11m52.778383625s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.255879032 (445h11m52.807349503s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.937012666 (445h11m53.488483147s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:55.385905965 (445h11m53.937376426s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:58.996130974 (445h11m57.547601435s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:02.960102785 (445h12m1.511573296s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:04.751312700 (445h12m3.302783161s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:05.874651098 (445h12m4.426121549s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:06.557657779 (445h12m5.10912824s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:18.597005313 (445h12m17.148475774s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:50930->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597177662 (445h12m17.148648123s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:40618->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597200885 (445h12m17.148671336s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:53296->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597348557 (445h12m17.148819018s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:36742->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597425313 (445h12m17.148895774s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: expected piece hash; context deadline exceeded; EOF Nov 27 21:16:18.597465990 (445h12m17.148936451s) [seg_pos={0 0}, upload=392027]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:16:18.842083808 (445h12m17.393554269s) [upload=392027]: Done uploading segments... Nov 27 21:16:18.842126799 (445h12m17.39359726s) [upload=392027]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 12.983591394s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.706039647s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:18:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.706069032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X.......X......XX.....................X..] duration: 22.031118475s trace: https://jaeger.dev.storj.io/trace/95534c84b640d45 FAIL (2025-11-27 21:18:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 21:15:49.300568539 (445h11m47.852039s) [upload=392027]: Starting upload Nov 27 21:15:49.300648992 (445h11m47.852119453s) [upload=392027]: Done starting upload Nov 27 21:15:49.300845096 (445h11m47.852315557s) [upload=392027]: Uploading segments... Nov 27 21:15:49.300959323 (445h11m47.852429774s) [upload=392027]: Got next segment. Inline: false Nov 27 21:15:49.301019478 (445h11m47.852489939s) [upload=392027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:15:49.301026451 (445h11m47.852496912s) [upload=392027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:15:49.547343071 (445h11m48.098813532s) [seg_pos={0 0}, upload=392027]: Begin upload segment... Nov 27 21:15:49.547402453 (445h11m48.098872914s) [seg_pos={0 0}, upload=392027]: Done begin upload segment. Nov 27 21:15:49.547441377 (445h11m48.098911838s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.547457709 (445h11m48.09892817s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548172770 (445h11m48.099643231s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548203278 (445h11m48.099673749s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548580657 (445h11m48.100051108s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548745050 (445h11m48.100215501s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548991479 (445h11m48.10046194s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549128791 (445h11m48.100599252s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549445283 (445h11m48.100915754s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550022723 (445h11m48.101493174s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550538054 (445h11m48.102008515s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550650238 (445h11m48.102120689s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551095967 (445h11m48.102566418s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551630003 (445h11m48.103100455s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551905909 (445h11m48.10337637s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.552366957 (445h11m48.103837418s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:51.295819242 (445h11m49.847289693s) [upload=392027]: Next returned nil segment Nov 27 21:15:51.295883825 (445h11m49.847354276s) [upload=392027]: Waiting for error group managing segments... Nov 27 21:15:51.607974413 (445h11m50.159444874s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:51.613231052 (445h11m50.164701514s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.226913164 (445h11m52.778383625s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.255879032 (445h11m52.807349503s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.937012666 (445h11m53.488483147s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:55.385905965 (445h11m53.937376426s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:58.996130974 (445h11m57.547601435s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:02.960102785 (445h12m1.511573296s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:04.751312700 (445h12m3.302783161s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:05.874651098 (445h12m4.426121549s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:06.557657779 (445h12m5.10912824s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:18.597005313 (445h12m17.148475774s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:50930->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597177662 (445h12m17.148648123s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:40618->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597200885 (445h12m17.148671336s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:53296->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597348557 (445h12m17.148819018s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:36742->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597425313 (445h12m17.148895774s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: expected piece hash; context deadline exceeded; EOF Nov 27 21:16:18.597465990 (445h12m17.148936451s) [seg_pos={0 0}, upload=392027]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:16:18.842083808 (445h12m17.393554269s) [upload=392027]: Done uploading segments... Nov 27 21:16:18.842126799 (445h12m17.39359726s) [upload=392027]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 22.031143292s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.97119573s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:18:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.971223743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X.......X......XX.....................X.] duration: 5.821374864s trace: https://jaeger.dev.storj.io/trace/95534c84b640d45 FAIL (2025-11-27 21:17:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 21:15:49.300568539 (445h11m47.852039s) [upload=392027]: Starting upload Nov 27 21:15:49.300648992 (445h11m47.852119453s) [upload=392027]: Done starting upload Nov 27 21:15:49.300845096 (445h11m47.852315557s) [upload=392027]: Uploading segments... Nov 27 21:15:49.300959323 (445h11m47.852429774s) [upload=392027]: Got next segment. Inline: false Nov 27 21:15:49.301019478 (445h11m47.852489939s) [upload=392027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:15:49.301026451 (445h11m47.852496912s) [upload=392027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:15:49.547343071 (445h11m48.098813532s) [seg_pos={0 0}, upload=392027]: Begin upload segment... Nov 27 21:15:49.547402453 (445h11m48.098872914s) [seg_pos={0 0}, upload=392027]: Done begin upload segment. Nov 27 21:15:49.547441377 (445h11m48.098911838s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.547457709 (445h11m48.09892817s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548172770 (445h11m48.099643231s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548203278 (445h11m48.099673749s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548580657 (445h11m48.100051108s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548745050 (445h11m48.100215501s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548991479 (445h11m48.10046194s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549128791 (445h11m48.100599252s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549445283 (445h11m48.100915754s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550022723 (445h11m48.101493174s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550538054 (445h11m48.102008515s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550650238 (445h11m48.102120689s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551095967 (445h11m48.102566418s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551630003 (445h11m48.103100455s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551905909 (445h11m48.10337637s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.552366957 (445h11m48.103837418s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:51.295819242 (445h11m49.847289693s) [upload=392027]: Next returned nil segment Nov 27 21:15:51.295883825 (445h11m49.847354276s) [upload=392027]: Waiting for error group managing segments... Nov 27 21:15:51.607974413 (445h11m50.159444874s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:51.613231052 (445h11m50.164701514s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.226913164 (445h11m52.778383625s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.255879032 (445h11m52.807349503s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.937012666 (445h11m53.488483147s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:55.385905965 (445h11m53.937376426s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:58.996130974 (445h11m57.547601435s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:02.960102785 (445h12m1.511573296s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:04.751312700 (445h12m3.302783161s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:05.874651098 (445h12m4.426121549s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:06.557657779 (445h12m5.10912824s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:18.597005313 (445h12m17.148475774s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:50930->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597177662 (445h12m17.148648123s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:40618->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597200885 (445h12m17.148671336s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:53296->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597348557 (445h12m17.148819018s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:36742->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597425313 (445h12m17.148895774s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: expected piece hash; context deadline exceeded; EOF Nov 27 21:16:18.597465990 (445h12m17.148936451s) [seg_pos={0 0}, upload=392027]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:16:18.842083808 (445h12m17.393554269s) [upload=392027]: Done uploading segments... Nov 27 21:16:18.842126799 (445h12m17.39359726s) [upload=392027]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.821435289s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.723198552s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:16:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.723221969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X.......X......XX.....................X] duration: 30.247243215s trace: https://jaeger.dev.storj.io/trace/95534c84b640d45 FAIL (2025-11-27 21:16:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 21:15:49.300568539 (445h11m47.852039s) [upload=392027]: Starting upload Nov 27 21:15:49.300648992 (445h11m47.852119453s) [upload=392027]: Done starting upload Nov 27 21:15:49.300845096 (445h11m47.852315557s) [upload=392027]: Uploading segments... Nov 27 21:15:49.300959323 (445h11m47.852429774s) [upload=392027]: Got next segment. Inline: false Nov 27 21:15:49.301019478 (445h11m47.852489939s) [upload=392027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:15:49.301026451 (445h11m47.852496912s) [upload=392027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:15:49.547343071 (445h11m48.098813532s) [seg_pos={0 0}, upload=392027]: Begin upload segment... Nov 27 21:15:49.547402453 (445h11m48.098872914s) [seg_pos={0 0}, upload=392027]: Done begin upload segment. Nov 27 21:15:49.547441377 (445h11m48.098911838s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.547457709 (445h11m48.09892817s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548172770 (445h11m48.099643231s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548203278 (445h11m48.099673749s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548580657 (445h11m48.100051108s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548745050 (445h11m48.100215501s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.548991479 (445h11m48.10046194s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549128791 (445h11m48.100599252s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.549445283 (445h11m48.100915754s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550022723 (445h11m48.101493174s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550538054 (445h11m48.102008515s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.550650238 (445h11m48.102120689s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551095967 (445h11m48.102566418s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551630003 (445h11m48.103100455s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.551905909 (445h11m48.10337637s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:49.552366957 (445h11m48.103837418s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Uploading piece... Nov 27 21:15:51.295819242 (445h11m49.847289693s) [upload=392027]: Next returned nil segment Nov 27 21:15:51.295883825 (445h11m49.847354276s) [upload=392027]: Waiting for error group managing segments... Nov 27 21:15:51.607974413 (445h11m50.159444874s) [noise=true , address=77.90.31.19:4842 , piece_id=2WL3D3BAZQKC7NZQHIEHB2LFJUIPLLJVT47IM7DNPPVBIVWMLZIA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:51.613231052 (445h11m50.164701514s) [noise=true , address=45.91.116.4:4940 , piece_id=GN4HJVYOGYNEETBXBZCVBNW2LGZ6NWM57TSGBS7YPOQTMN5VZOBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.226913164 (445h11m52.778383625s) [noise=true , address=145.79.227.14:4840 , piece_id=DJIGEQ4RTZ5OPHZZF6C3VL5XFBWN5GEVALA6VKVREJVH425ZTRQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.255879032 (445h11m52.807349503s) [noise=true , address=145.79.227.43:4159 , piece_id=GCCQQFBCTITRM7Z7GHUWUO7DKXPDDMVQXL7TUU3D7UKFDLZTSQBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:54.937012666 (445h11m53.488483147s) [noise=true , address=145.79.227.22:4511 , piece_id=E6Y3MRR2ZBEAJILPGRLBSF5IIAZGEUCSQLGZ4CIYW3F7IOXY74ZQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:55.385905965 (445h11m53.937376426s) [noise=true , address=145.79.227.30:4761 , piece_id=VAPDQZDMYCAMN2JF23NJT6NYYNA2VSTGHJOV4QBTHA4XQ2YPOTQQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:15:58.996130974 (445h11m57.547601435s) [noise=true , address=145.79.227.8:4329 , piece_id=4VWZ3XCKUYIVXUQ7EFXP4O6OWPC5DOB2LHCFE4F74IJ77DMXXSVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:02.960102785 (445h12m1.511573296s) [noise=true , address=145.79.227.9:4367 , piece_id=VZ5C6633TQCBLHWPPDGEUIUEKXFPCXE4YAUD2TE7X7KP42P7DVEQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:04.751312700 (445h12m3.302783161s) [noise=true , address=145.79.227.16:4909 , piece_id=SDDCGPCBKDYPG22EKBUEIXQMUYABZG2ZW3UNI3JOLIF2H5YM6PVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:05.874651098 (445h12m4.426121549s) [noise=true , address=145.79.227.43:4158 , piece_id=3PK64JGBO3BXVQQE3ARDF6ATDJ7GPCB4R6GJFHTOFZRYT324UZVA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:06.557657779 (445h12m5.10912824s) [noise=true , address=145.79.227.13:4804 , piece_id=32U3TIVPIUZWVPYCNLQSPUAC2JVP2H4PNMELSXPUXVESXLXI66XQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: <nil> Nov 27 21:16:18.597005313 (445h12m17.148475774s) [noise=true , address=145.79.227.21:4473 , piece_id=KR27WR3EUCY7ZJWUXPEZWPACWXSEDIGMYX7OOBCNSPJSUXAJR4GA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:50930->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597177662 (445h12m17.148648123s) [noise=true , address=145.79.227.29:4943 , piece_id=IPDRJUXMB5W5PC7GFUGF4Y7RYYK53QZQKP7URVCV6U6B5EFBTSBQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:40618->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597200885 (445h12m17.148671336s) [noise=true , address=145.79.227.6:4258 , piece_id=WCPJUX7PYJ4GRANNMY76EHCAZHTGBRJW2KOU2XNRYXOSP4CMCOWA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:53296->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597348557 (445h12m17.148819018s) [noise=true , address=145.79.227.39:4014 , piece_id=Z57WMWJFCL3KQZTQQDCNLTWLH2YWFKDTRS6Z5CC4SGMO3ANZODDA, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:36742->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:16:18.597425313 (445h12m17.148895774s) [noise=true , address=145.79.227.41:4087 , piece_id=NINW2V6R67IS5B6EQR7CRKRQ2GYOFUYEK2LR3P6QOLDYI6FFNIIQ, seg_pos={0 0}, upload=392027]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: expected piece hash; context deadline exceeded; EOF Nov 27 21:16:18.597465990 (445h12m17.148936451s) [seg_pos={0 0}, upload=392027]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 21:16:18.842083808 (445h12m17.393554269s) [upload=392027]: Done uploading segments... Nov 27 21:16:18.842126799 (445h12m17.39359726s) [upload=392027]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.247267481s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.037751472s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:16:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.037770088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.......X......XX.....................] duration: 5.780506095s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:15:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.780600827s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.185728336s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:15:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.185754345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.......X......XX....................] duration: 5.836454688s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:14:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.836568586s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.740482868s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:13:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.740508657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X.......X......XX...................] duration: 17.297023045s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:13:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 17.297080403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........................X.......X......XX..................] duration: 5.77364344s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:13:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.77367472s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.130181693s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:13:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.130209436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........................X.......X......XX.................] duration: 5.815284673s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:12:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.815314088s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.710645193s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:12:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.711256997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........................X.......X......XX................] duration: 5.727315355s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:11:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.727345652s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.908009714s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:11:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.908036845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X........................X.......X......XX...............] duration: 5.669501305s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:10:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.669532993s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.901854106s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:10:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.901886828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X........................X.......X......XX..............] duration: 5.660719078s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:10:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.660753405s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.797917514s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:09:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.79794671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X........................X.......X......XX.............] duration: 5.986950138s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:09:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.986980907s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.026322908s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:08:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.026398121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X........................X.......X......XX............] duration: 5.795671391s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:08:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.795699854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X........................X.......X......XX...........] duration: 5.798364885s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:08:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.798397397s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.759361936s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:08:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.759422121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X........................X.......X......XX..........] duration: 5.67498822s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:07:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.675019158s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.849460721s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:07:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.849492722s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.94017299s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:06:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.940202515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X........................X.......X......XX.........] duration: 5.833774074s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:06:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.834094895s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.994345656s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:05:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.994391083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X........................X.......X......XX........] duration: 5.90349706s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:05:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.903523892s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.975350256s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:04:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.975377668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X........................X.......X......XX.......] duration: 5.721598657s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:04:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.721646378s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.565223365s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:03:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.565250888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X........................X.......X......XX......] duration: 5.839094603s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:03:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.839116967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X............X........................X.......X......XX.....] duration: 5.746462413s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:02:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.74650245s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.795567057s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:02:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.795586644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X............X........................X.......X......XX....] duration: 7.298669429s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:02:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 7.298699426s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.996975157s trace: https://jaeger.dev.storj.io/trace/5038ea467d0a3c9b FAIL (2025-11-27 21:01:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.997001749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X............X........................X.......X......XX...] duration: 5.826117443s trace: https://jaeger.dev.storj.io/trace/44462f8c8e937986 FAIL (2025-11-27 21:01:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== em_ObjectBegin Nov 27 20:59:09.000354615 (444h55m7.551825076s) [upload=391790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:59:09.264698970 (444h55m7.816169431s) [seg_pos={0 0}, upload=391790]: Begin upload segment... Nov 27 20:59:09.264751651 (444h55m7.816222192s) [seg_pos={0 0}, upload=391790]: Done begin upload segment. Nov 27 20:59:09.264794212 (444h55m7.816264673s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.264854477 (444h55m7.816324938s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265436885 (444h55m7.816907336s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265574607 (444h55m7.817045068s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.265977024 (444h55m7.817447475s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.266446809 (444h55m7.81791727s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271170523 (444h55m7.822640974s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.271184671 (444h55m7.822655132s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.272651263 (444h55m7.824121714s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.273440415 (444h55m7.824910876s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.274075565 (444h55m7.825546016s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.275516689 (444h55m7.82698715s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.276812336 (444h55m7.828282787s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.277537036 (444h55m7.829007507s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.278328754 (444h55m7.829799205s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:09.279735654 (444h55m7.831206105s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Uploading piece... Nov 27 20:59:11.052229614 (444h55m9.603700065s) [upload=391790]: Next returned nil segment Nov 27 20:59:11.052247138 (444h55m9.603717599s) [upload=391790]: Waiting for error group managing segments... Nov 27 20:59:11.334160139 (444h55m9.8856306s) [noise=true , address=77.90.31.10:4520 , piece_id=UWRZI7KO3BGIQK63PFXHWJA52NJOB7M2CUZP2KEUBHG36ZIMALXQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:29.774036574 (444h55m28.325507035s) [noise=true , address=145.79.227.30:4761 , piece_id=7SLVN6B2TZANQLXP4JZO4AAPHDPJMSL7VZSFV2PTWQ2NZ3I2YULA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.234303969 (444h55m28.78577443s) [noise=true , address=145.79.227.17:4971 , piece_id=PWQOQFIBR3222NO2OUVSZVF5NB7V4QDALTC5SF4EXJZJOBFMN6HQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:30.790680203 (444h55m29.342150664s) [noise=true , address=145.79.227.43:4158 , piece_id=V64JCFE7IIDR4LZB7KPBAJSLNUHLRL6L3UVPHLJXSIRIODEUMNZA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:31.082922793 (444h55m29.634393254s) [noise=true , address=145.79.227.20:4437 , piece_id=7N52CDZULMCLVWJ3PMSEQT2SSTGRFX2CJ4JFCJLJC6VSEWBKTPBQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:32.895242502 (444h55m31.446712953s) [noise=true , address=145.79.227.33:4869 , piece_id=JI7T4KIUYSP3PC7ZOOYW3WTUSRIAKRHT3CFFYVI3C5OLMTCMMFCQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:35.028015659 (444h55m33.57948612s) [noise=true , address=145.79.227.24:4583 , piece_id=KF3F7AAHOJC5JSRMA4TXZEEFZZPDZ2FX2EGZKA45E73FM4WAFTRA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: <nil> Nov 27 20:59:38.247946801 (444h55m36.799417252s) [noise=true , address=145.79.227.19:4079 , piece_id=O6FXW6VE7YB6OUGKHEB6RB7NWITNSFQEJWTWA4AEFOHQG6DQOIIQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248000664 (444h55m36.799471125s) [noise=true , address=145.79.227.39:4013 , piece_id=2CVZSFW4G2MQCPY7SF2THGZ6U4DTFB6NUVAMLRGG4J2W2VIGZDSQ, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: rpc: dial tcp 145.79.227.39:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248176799 (444h55m36.79964725s) [noise=true , address=145.79.227.8:4331 , piece_id=N5NLOTWJ7NDPKBBXMCZVZVYJCXIQE5UYW7LPN54JK6CIRMQ4D5WA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248234308 (444h55m36.799704769s) [noise=true , address=145.79.227.8:4329 , piece_id=UCY4BNECQT2DFYK7R57CH6NS66VLR7PDJIX2KP2QFPEH36J57QVA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35048->145.79.227.8:4329: use of closed network connection Nov 27 20:59:38.248396387 (444h55m36.799866838s) [noise=true , address=145.79.227.10:4403 , piece_id=NUF2FDOJJAO4FQDPKMH6UCDDJINE4KTLG6ZR5OX5A3Y43FBHETFA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: rpc: dial tcp 145.79.227.10:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248527797 (444h55m36.799998248s) [noise=true , address=145.79.227.14:4838 , piece_id=ZSB7MGM6GVMFEVFVTP3ONNDCVX2WXU3ASKV5VIRA6QIYTIV55J7Q, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248665860 (444h55m36.800136321s) [noise=true , address=145.79.227.41:4086 , piece_id=HG6FZN6TTFFJ7S5IS23F2Z5K7D2LIGGLFLSSLLUG4EW7MYY7SP7A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: rpc: dial tcp 145.79.227.41:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.248965801 (444h55m36.800436262s) [noise=true , address=145.79.227.11:4730 , piece_id=ZUAMIPUESIWD5T3CEZLDZUQQI4FGXWG4US23O26NIUYVX7ICL44A, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249111148 (444h55m36.800581609s) [noise=true , address=145.79.227.16:4909 , piece_id=CILAOKAIJ5PKK257NPIGUBIWAZOJB4E3EG2KYRIN6IR6PPD2IMKA, seg_pos={0 0}, upload=391790]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:47890->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:59:38.249147357 (444h55m36.800617808s) [seg_pos={0 0}, upload=391790]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 27 20:59:38.497730300 (444h55m37.049200761s) [upload=391790]: Done uploading segments... Nov 27 20:59:38.497773562 (444h55m37.049244023s) [upload=391790]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.826145527s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)