Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.937097453s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:06:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.937223983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.932414939s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.932489892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.771868307s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 05:05:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.771921618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.043597793s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.043621458s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.702357179s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 05:04:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.702397496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.922790271s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:04:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.922824516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.913214826s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:03:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.913243972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.80362252s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 05:03:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.803814855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.09945982s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:02:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.0995226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.807759992s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 05:02:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.807787634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.759490986s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 05:02:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.759561991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.047278303s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:02:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.04732442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.919172592s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:01:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.919191979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.982432794s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:01:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.982465407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.79301363s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 05:01:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.793040512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.132290278s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.132317409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.83331359s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 05:00:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.833341732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.034851466s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 05:00:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.034882746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.152834633s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 05:00:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.152868518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.023387126s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:59:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.023419248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.879889182s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:58:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.879920211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.906727788s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:58:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.906755342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.928022451s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:58:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.92805347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.772759882s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:58:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.772787184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.993142051s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:57:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.993173311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.640953239s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:57:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.640976952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.986543946s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:57:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.986574114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.719869102s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:56:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.719891132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.850968151s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:56:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.850995233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.988530226s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:55:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.988577536s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.723755637s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:55:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.723877539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.786605461s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:55:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.786640176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.893213886s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:55:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.893240197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.093098096s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:54:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.093125378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.758700683s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:54:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.758733465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.865938037s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:53:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.865964086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.942042476s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:53:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.942072114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.91702735s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:53:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.917054662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.89940753s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:52:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.89942387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.058141574s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:52:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.058168524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.910139177s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:52:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.910166911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.808134402s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:51:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.808161253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.914068492s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:51:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.914100523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.934423168s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:51:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.934449688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.860336781s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:51:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.860353584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.053933301s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:50:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.053963318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.565025868s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:50:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.565054964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.620086254s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:50:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.620109258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.75717014s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:49:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.757200628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.081409812s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:49:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.081443759s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)