Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.891174973s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:09:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.891202085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.429550583s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:08:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429566303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.804633624s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:08:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.804662509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.37760634s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:07:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377647478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.97924748s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:07:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.979277689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.079499039s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:06:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.079524218s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.525928524s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:06:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525956225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.075628483s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:06:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.075656076s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.505460351s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:05:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505493325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 7.502067521s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:05:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 7.502113358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.180259205s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:04:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.18028891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 4.251310783s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:04:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.25134065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.934687602s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:04:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.934709042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.98970804s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:03:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.989739089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.473426958s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:03:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47345391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.056357383s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:03:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.056451903s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.449450094s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:02:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449482295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.02549702s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.025527679s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.494216618s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494261243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.082659509s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:01:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.082688153s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.410971082s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:01:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.410990849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.506255612s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:00:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507512975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.09613056s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 05:00:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.096157009s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.617262493s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 05:00:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.617290816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.834721797s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:59:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.834754138s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.371707828s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:58:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.371736192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.822271169s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:58:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.822302358s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.475238728s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:58:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475264066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.81615916s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:57:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.816186382s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 4.508630405s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:57:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.508661895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.031968762s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:57:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.032001316s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.520244617s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:57:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520273123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.437099088s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:56:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437661758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.892954585s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.892981606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.495283742s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:55:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495310211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.174339896s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:55:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.174370553s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.407959479s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:54:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407995367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.842454195s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:54:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.842486196s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.438957453s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:53:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439020924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.318602129s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:53:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.318637346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.523028983s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:52:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523103276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.650329852s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:52:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.650370869s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.612573677s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:52:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.612608343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.301104261s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:52:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.301135321s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.481120571s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:51:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481154125s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.530836919s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:51:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53091049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.911596159s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:51:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.91162303s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.508611377s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:50:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508636525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.438903681s trace: https://jaeger.dev.storj.io/trace/709d4b878ab52faf FAIL (2025-11-16 04:50:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438938857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.021512618s trace: https://jaeger.dev.storj.io/trace/530212e0902dafb9 FAIL (2025-11-16 04:50:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [seg_pos={0 0}, upload=144830]: Done begin upload segment. Nov 16 04:34:33.905874485 (164h30m32.457344936s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.905917137 (164h30m32.457387588s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906353367 (164h30m32.457823828s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.906791702 (164h30m32.458262153s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907143231 (164h30m32.458613682s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907504719 (164h30m32.45897518s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.907900554 (164h30m32.459371015s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908319121 (164h30m32.459789572s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.908988465 (164h30m32.460458926s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.910017283 (164h30m32.461487744s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.912748364 (164h30m32.464218855s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.914578489 (164h30m32.46604895s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.917073800 (164h30m32.468544262s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.919149083 (164h30m32.470619534s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.922225560 (164h30m32.473696021s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:33.924908239 (164h30m32.4763787s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Uploading piece... Nov 16 04:34:35.687735598 (164h30m34.239206059s) [upload=144830]: Next returned nil segment Nov 16 04:34:35.687751568 (164h30m34.239222029s) [upload=144830]: Waiting for error group managing segments... Nov 16 04:34:36.005926974 (164h30m34.557397426s) [noise=true , address=77.90.31.16:4735 , piece_id=QLDIJSDE5NRJNCJYN2ES2BF2ZPI4B2QVH4K3GP6R2QRO5ULU6C6A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.039609111 (164h30m34.591079563s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWFR62P2X7GWV7G37GTVE42HGVNNE6WXPZDNH4PBBSHXLKQ7QQQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.052952530 (164h30m34.604422991s) [noise=true , address=45.91.116.8:4047 , piece_id=ZXD75M2LUBCGECDCABCHYOYT6URHJWNJNVMU6TFUT7PJZOOTDMOQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.077267220 (164h30m34.628737671s) [noise=true , address=77.90.31.13:4628 , piece_id=QY4AQVMHCXH6WM7355JVJGV33GMJWQRYK6MQM3NDWPCLONVECSSQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.186309467 (164h30m34.737779928s) [noise=true , address=45.91.116.13:4154 , piece_id=DZUV6A2AXPGTC4JXMD5MCGYAPS4HXHAP2RAEQFVCBTHZ3G5MNJBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.626168172 (164h30m35.177638633s) [noise=true , address=145.79.227.11:4732 , piece_id=R2AUXFJ75JDTK3B2QL6N7JRKDCHPZECI37ECM4KOZ72RZB5ORZ4A, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:34:36.718121479 (164h30m35.26959194s) [noise=true , address=145.79.227.21:4474 , piece_id=UDI5SUERVAFEB56N5GGNCSD7LDKHLGLIBHKOY4AX7IMEYK2BYSYA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: <nil> Nov 16 04:35:02.941441015 (164h31m1.492911506s) [noise=true , address=145.79.227.9:4365 , piece_id=33MZWT24SAEE2HQXYC32NYYDCRDT4QPHI3RVIUVNBXYWJNBXMMAA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:41968->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941592152 (164h31m1.493062624s) [noise=true , address=145.79.227.27:4690 , piece_id=GSZGGECVXLEXNTO62UAWWFP5AFZKZKNM6JBS6VSQIDXZJSYK75OQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:46144->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.941621699 (164h31m1.49309216s) [noise=true , address=145.79.227.13:4803 , piece_id=7A7O2EO5O7DRDB4YNYASQDGQQHTX6HAMJE7ABCTXGYCXWEXJNCBA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->145.79.227.13:4803: use of closed network connection Nov 16 04:35:02.942000350 (164h31m1.493470821s) [noise=true , address=145.79.227.28:4727 , piece_id=D5B27U6HKR6FISQVZOO4IIPJZOH6OUNQRSSXQQ5C3T6GKEZDW7XA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:50584->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942139164 (164h31m1.493609615s) [noise=true , address=145.79.227.31:4798 , piece_id=SC42UOKXNUSJQOTPNHEQNNWQR72YI4M26NNNR4YOUL5CSBTN5PCA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: read tcp 188.166.225.170:36818->145.79.227.31:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942206994 (164h31m1.493677455s) [noise=true , address=145.79.227.25:4619 , piece_id=YGRAXPADTQTOUYUWJHAQ5UT2CVDRDA3U3CA2STNCLSQYTRZML5YA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:56844->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942326631 (164h31m1.493797082s) [noise=true , address=145.79.227.10:4401 , piece_id=R5U4OIE5Y2QF6R2AJ4ULKHIZLWKMAIYVEKJYUIPHNBO22GR6KKOA, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: read tcp 188.166.225.170:48428->145.79.227.10:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942421953 (164h31m1.493892414s) [noise=true , address=145.79.227.16:4909 , piece_id=PEAVPGZMWPMGSLC7WTKLSGQRQZ3RYKGNM5RDN4XO7UBWU5GD4SUQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:34450->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942512055 (164h31m1.493982516s) [noise=true , address=145.79.227.30:4763 , piece_id=JTBWLNP6Q3LVJV5IBK2SCCCDWQ52RFAZOG7Y2RXJZ5DWQO5EYSMQ, seg_pos={0 0}, upload=144830]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:60934->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 04:35:02.942546570 (164h31m1.494017042s) [seg_pos={0 0}, upload=144830]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 16 04:35:03.198619431 (164h31m1.750089892s) [upload=144830]: Done uploading segments... Nov 16 04:35:03.198671089 (164h31m1.75014155s) [upload=144830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.021551391s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)