Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 5.791145842s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:40:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.791199686s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 5.669004653s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:39:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.669032686s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 7.075047566s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:38:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 7.075079847s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 5.36596049s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:37:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.365990757s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 5.746087939s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:37:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.746122265s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 5.6772864s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:36:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.67731788s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 5.585160934s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:35:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.585192124s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 5.371082945s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:35:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.371108294s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 5.750998936s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:34:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.751030356s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 5.421777581s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:34:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.421885056s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 5.810729522s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:33:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.810750632s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 5.415508504s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:32:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.415535856s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 5.750062375s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:32:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 5.750089747s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 1m0.008792213s trace: https://jaeger.dev.storj.io/trace/3205ce1428ca94c1 FAIL (2025-11-16 17:32:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267219259 (177h27m0.81868973s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.267848107 (177h27m0.819318568s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.268531769 (177h27m0.82000223s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269170986 (177h27m0.820641447s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269210201 (177h27m0.820680662s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.269842595 (177h27m0.821313056s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.270489707 (177h27m0.821960168s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271183227 (177h27m0.822653689s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.271889012 (177h27m0.823359473s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:02.274388021 (177h27m0.825858492s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Uploading piece... Nov 16 17:31:03.622120670 (177h27m2.173591131s) [upload=156289]: Next returned nil segment Nov 16 17:31:03.622141931 (177h27m2.173612392s) [upload=156289]: Waiting for error group managing segments... Nov 16 17:31:04.161830843 (177h27m2.713301314s) [noise=true , address=185.167.204.205:4022 , piece_id=HLSZY3C344MMOC7UQYZ7DWBL6XLYUUBYSWDCRDUYOJ75XA4P5LBQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.177200319 (177h27m2.72867078s) [noise=true , address=185.167.204.206:4045 , piece_id=2XGWBL7OIDTRAMP5GQIXE5UTFKAP2E75MKISSU4CN75XKUK5SYNA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188379868 (177h27m2.739850349s) [noise=true , address=185.167.204.206:4050 , piece_id=M3AMQZBPFY7YBQRYCILU6ZNQQJLXJX6H73JHZD5SVFWYIHUZHEBA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.188402411 (177h27m2.739872872s) [noise=true , address=185.167.204.205:4005 , piece_id=GFCIJLGT3KKKB44UO5KSUDJIVOGRJPSD7ATRY6ILNJLVJOPCHQEQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196455605 (177h27m2.747926076s) [noise=true , address=185.167.204.205:4004 , piece_id=CQF5Z5SSYWPGTPIYSOYLIACHYQ5KS2BFHHS7QIHZX44PAZY5FJKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.196572709 (177h27m2.74804317s) [noise=true , address=185.167.204.205:4028 , piece_id=D3JFSDDUQPLGSFIIFVZLNQUQYBLHV2QEJRJEXYWWCRETPFPRVCJA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.199515673 (177h27m2.750986134s) [noise=true , address=185.167.204.206:4072 , piece_id=KFUF2F3PF6FHEZV4XRJUS3AI44ANRYANVGMRAAMJLISA7ISVVQ6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.201356709 (177h27m2.75282717s) [noise=true , address=185.167.204.206:4041 , piece_id=KCML6QMARV3SRIPW265ZUG2HYNQCDNGKLICC6BYCLBSK6TPFNG5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.207006307 (177h27m2.758476768s) [noise=true , address=185.167.204.206:4069 , piece_id=RTWHXVVALH4WUOHBOCPJ25Y3TJSOZIR5BZ3EEVPE5FGNUOEUPGEA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.229709449 (177h27m2.78117991s) [noise=true , address=185.167.204.206:4039 , piece_id=5TNNNVPHSEKLKBNTG3GKED2ACCJBBNRQLWVJVNTOQW3GT46NNE4Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.230896139 (177h27m2.78236659s) [noise=true , address=185.167.204.205:4027 , piece_id=6TVE3BOAR53VN6IZSXXDIHNYPYLPXITNCYHIGZK7UH6M22JODW6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.668503714 (177h27m3.219974175s) [noise=true , address=185.167.204.205:4016 , piece_id=4EZHFO77KZU5EHAG22WVWESCHP7CC6NP7C2TTXDHXXS53T4P4RCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.731733407 (177h27m3.283203878s) [noise=true , address=89.149.201.147:4009 , piece_id=YIATD6EFCM5U3YI2WWYGHYTLKQNXXHHMC7CG2SWB4R3Y2SXP4TYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.786581410 (177h27m3.338051861s) [noise=true , address=89.149.201.147:4016 , piece_id=WJJMDHEYVHZEH426X2TK6AJNVM6YQ5K5BHMIOPWA5HZA6UEOGRLQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.977188750 (177h27m3.528659211s) [noise=true , address=185.167.204.206:4067 , piece_id=UGYZTD7MQPPI7EPUOT56YFVY456LK2O57YQEWT75JQSCLBAY6UCA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:04.986870056 (177h27m3.538340517s) [noise=true , address=185.167.204.205:4023 , piece_id=Q3DJS6GQ4K6PLI4UY6DHV7GDUNPYVSMNCDAWR4FKEAQPY4XCQOSA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.043186063 (177h27m3.594656525s) [noise=true , address=185.167.204.205:4010 , piece_id=EEWFQBRLYMUHASVMYFVG7IT4TEE75IB4OE63VKR7DGX4ODJHGYTA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.087234302 (177h27m3.638704763s) [noise=true , address=185.167.204.206:4052 , piece_id=KUM5O3P24LJRUD6XDR7BHN5GNTZIRGFS4THCHTATRZIITIPIQYYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:05.314404745 (177h27m3.865875206s) [noise=true , address=89.149.201.147:4011 , piece_id=BKAP6XL25MW5R7IBQZ5P6JXBP7OOO6TXMSJU5CR675UWXDIQFDAA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:07.459378486 (177h27m6.010848947s) [noise=true , address=81.171.14.70:4030 , piece_id=AWL2QXNDEIAUUWFKRDYZR4CTQCLQH5APKQPRMI4JYGZLQWXALC5Q, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:21.997392585 (177h27m20.548863046s) [noise=true , address=81.171.14.70:4040 , piece_id=PR4GJQFOJS7MNU2U3P7TEOTXMAJDCOS7PPVDXBEYQVRL4NERFSYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:25.424594109 (177h27m23.97606457s) [noise=true , address=81.171.14.70:4048 , piece_id=6XXKRR4UKN2TWYCKC24SYFNJWOBW7BACGB3R5MKO46OBESUHKDYA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:27.583739904 (177h27m26.135210355s) [noise=true , address=81.171.14.70:4042 , piece_id=HJIZQLK6UAAME6M5IWUJN64VXH7UXFFNKPGWESCXB5EQMASN4CKA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.366399766 (177h27m34.917870227s) [noise=true , address=81.171.14.70:4035 , piece_id=S6KAHK4LE747WX35VMJ7ZI53V6ZTJWTQ77XB5BAEZ3BNAUPUE7CQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:36.620762982 (177h27m35.172233483s) [noise=true , address=81.171.14.70:4026 , piece_id=EENDSDICS3KDI2LOT2FTZHER2YU3ZAL2NIRNFHNUXYPDIX4I2NCQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925261239 (177h27m58.4767317s) [noise=true , address=81.171.14.70:4045 , piece_id=3SMH6SKS24FUQRTGRHLJUNAG5CQDPNGRZVCWKG33PVY6TMZILX6A, seg_pos={0 0}, upload=156289]: Done uploading piece. err: <nil> Nov 16 17:31:59.925293931 (177h27m58.476764402s) [seg_pos={0 0}, upload=156289]: Segment reached optimal threshold of 26 pieces. Nov 16 17:31:59.925547494 (177h27m58.477017955s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.925557913 (177h27m58.477028374s) [noise=true , address=81.171.14.70:4047 , piece_id=L7GMVM4IOMFWFFHSHZLOACRUOWZMCCHU22RUTBQLY7B5V2FYWBIA, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926178866 (177h27m58.477649327s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Nov 16 17:31:59.926190298 (177h27m58.477660759s) [noise=true , address=81.171.14.70:4046 , piece_id=TLQ3SPU4TOKQJMU5VP6T6D2SWUY2454ZG34RYSRWJTHFSJDEX5OQ, seg_pos={0 0}, upload=156289]: Long tail cancellation due to optimal threshold Nov 16 17:31:59.926348399 (177h27m58.47781886s) [seg_pos={0 0}, upload=156289]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 16 17:31:59.926461554 (177h27m58.477932015s) [upload=156289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 17:31:59.926469440 (177h27m58.477939901s) [upload=156289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 17:32:00.162960392 (177h27m58.714430853s) [upload=156289]: Done uploading segments... Nov 16 17:32:00.162982725 (177h27m58.714453176s) [upload=156289]: Upload finished. err: <nil> Duration: 1m0.00882173s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.500949586s trace: https://jaeger.dev.storj.io/trace/5a8411d863de3536 FAIL (2025-11-16 17:30:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== YZZQ2NOS2SXOCD76P6XFQUJ64ZRCID7X63CJKOUPQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.293883180 (177h25m2.845353641s) [noise=true , address=81.171.14.70:4044 , piece_id=GOJ5FCGXG76SD25KRISU23ZW7G6P5ZAKRZLDNWKIFJKBZQVCKDHQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.298949508 (177h25m2.850419999s) [noise=true , address=185.167.204.205:4023 , piece_id=XN2QSJ3V2E6IPJNLBEZVHG62A63DRN2ZHIL6YJVSQC2WMY5ELUFQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.299688725 (177h25m2.851159176s) [noise=true , address=89.149.201.147:4002 , piece_id=LKPZVTUHJEBMD4PMCWWQ77XK4EDUINAXTWZUD2CTCZL2MUMZGCHQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.300482637 (177h25m2.851953098s) [noise=true , address=89.149.201.147:4006 , piece_id=HE2XG5ITH2MRJQ6HPKH2TDBKVJY7XWTGVJNNL7K3LNFMMIWEDK2Q, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.301338105 (177h25m2.852808566s) [noise=true , address=81.171.14.70:4042 , piece_id=L3EX3TO6HZS3PNL5BQRN2NHMEEVBOF23DNESSZZZA6XWJ4OX5F5Q, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.301949750 (177h25m2.853420211s) [noise=true , address=89.149.201.147:4001 , piece_id=L5JVJICBH3RTDL5IUKWSDYLD4FYXKIBCUMLBI3ME7SREE3UG6SYQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.302735878 (177h25m2.854206339s) [noise=true , address=81.171.14.70:4033 , piece_id=SUI7KRZEHAAOQBXAVEK2Y3J227LSD2APRZNMXODQYT3H6FMB2DXA, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.304429843 (177h25m2.855900304s) [noise=true , address=81.171.14.70:4030 , piece_id=XQG75D4FPC5BPZUGMVE477FEPONHVBVEJSPQUHPW5QWCW3WECFEQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.306171960 (177h25m2.857642421s) [noise=true , address=185.167.204.205:4005 , piece_id=I7TKD7PUNFYSXEEQUOJATJSZNY7CJKU6VEAMAX26WHXED4GP732A, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.307777438 (177h25m2.859247899s) [noise=true , address=185.167.204.206:4039 , piece_id=PDZNLXGDXOUIAWDS2KTGVEJB5UT2TMBGK5AO4XYKHUEFW2TN2PDQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.310072839 (177h25m2.8615433s) [noise=true , address=185.167.204.205:4020 , piece_id=ZHEONHQSAGESWXZDUTKJMY4RHLPZUOV4KMZZL46H57XH2PRZG3UA, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:05.660189389 (177h25m4.21165985s) [upload=156257]: Next returned nil segment Nov 16 17:29:05.660208655 (177h25m4.211679126s) [upload=156257]: Waiting for error group managing segments... Nov 16 17:29:06.248075107 (177h25m4.799545568s) [noise=true , address=185.167.204.205:4027 , piece_id=46UM2RN7MTH6E224VUHCGSQ5ENQNRASNRHI642FOQJB2GR4D3PFA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.289496302 (177h25m4.840966763s) [noise=true , address=185.167.204.206:4039 , piece_id=PDZNLXGDXOUIAWDS2KTGVEJB5UT2TMBGK5AO4XYKHUEFW2TN2PDQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.399274525 (177h25m4.950744986s) [noise=true , address=185.167.204.205:4023 , piece_id=XN2QSJ3V2E6IPJNLBEZVHG62A63DRN2ZHIL6YJVSQC2WMY5ELUFQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.435677385 (177h25m4.987147836s) [noise=true , address=185.167.204.206:4069 , piece_id=HPPXP5DQIBTYZZQ2NOS2SXOCD76P6XFQUJ64ZRCID7X63CJKOUPQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.770064945 (177h25m5.321535406s) [noise=true , address=89.149.201.147:4001 , piece_id=L5JVJICBH3RTDL5IUKWSDYLD4FYXKIBCUMLBI3ME7SREE3UG6SYQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.776862789 (177h25m5.32833325s) [noise=true , address=89.149.201.147:4006 , piece_id=HE2XG5ITH2MRJQ6HPKH2TDBKVJY7XWTGVJNNL7K3LNFMMIWEDK2Q, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.782410283 (177h25m5.333880744s) [noise=true , address=89.149.201.147:4005 , piece_id=JVHBZ7N52CN2NBEBQJ3XP3W57PILUBBISMYCAZVDRB55G4JUEKSA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.814083125 (177h25m5.365553586s) [noise=true , address=89.149.201.147:4008 , piece_id=V4JRN3IPIXP2S2SYWV4GB2TWDGHS4PW66DCH2TTQOZLSDJMZJDDQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.868520486 (177h25m5.419990947s) [noise=true , address=185.167.204.206:4059 , piece_id=M5DQ3WJHV7AOSCFCJ442RVDOZP66KB3SP6GHD4JZGOZS7736SYRA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.900967262 (177h25m5.452437723s) [noise=true , address=89.149.201.147:4002 , piece_id=LKPZVTUHJEBMD4PMCWWQ77XK4EDUINAXTWZUD2CTCZL2MUMZGCHQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.959077197 (177h25m5.510547658s) [noise=true , address=185.167.204.205:4020 , piece_id=ZHEONHQSAGESWXZDUTKJMY4RHLPZUOV4KMZZL46H57XH2PRZG3UA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:07.043974090 (177h25m5.595444551s) [noise=true , address=185.167.204.205:4005 , piece_id=I7TKD7PUNFYSXEEQUOJATJSZNY7CJKU6VEAMAX26WHXED4GP732A, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:07.317941797 (177h25m5.869412258s) [noise=true , address=89.149.201.147:4014 , piece_id=OAXZ4DISEOHYICZEJAYMZYRTAUTE4OAZGQY26AXNBLAANCIKRU5Q, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:07.941429550 (177h25m6.492900001s) [noise=true , address=185.167.204.206:4064 , piece_id=NGINKCDVHB3FF2IVF3UAYAHCEXCAPOMG52DWAJCSX3I6ROFQE7XQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:08.384861638 (177h25m6.936332099s) [noise=true , address=185.167.204.206:4060 , piece_id=LMLIYFNLFFXITGXS2AC56EIUEURCWZQ4L2WZPAHQ3NRL5QWTMAIA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:09.438981884 (177h25m7.990452345s) [noise=true , address=185.167.204.206:4061 , piece_id=NXJMECD2HAI7DFLCDYRUMI2I5M3WL4U726LJMFRFILX4UJESFXQA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:10.820326839 (177h25m9.3717973s) [noise=true , address=185.167.204.206:4050 , piece_id=XLP3XGNM6GJKARQL4G7AEEIGLV23V5EX7KBZQYXU73WOKZ4R7NIA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:12.299104339 (177h25m10.8505748s) [noise=true , address=185.167.204.205:4004 , piece_id=NQSAX5TNX5SUVGAN5UQ6TCBGW2FJLUEROMTYLGGJWUG3B7ZJDVGA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:28.942804843 (177h25m27.494275304s) [noise=true , address=81.171.14.70:4028 , piece_id=7MVRUM3L57BAYTF6FJUQC5FEHCMXK4ULFLOURXLE5WNLLPYOX7OA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:37.084310989 (177h25m35.63578145s) [noise=true , address=81.171.14.70:4025 , piece_id=23BMULBLFPNYHAMH2YNVMV7SQSJOOWJRMG7PR5LBKW3GCDULQE4A, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:49.339948857 (177h25m47.891419318s) [noise=true , address=81.171.14.70:4030 , piece_id=XQG75D4FPC5BPZUGMVE477FEPONHVBVEJSPQUHPW5QWCW3WECFEQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:57.872400335 (177h25m56.423870786s) [noise=true , address=81.171.14.70:4040 , piece_id=IYDSHR5WIHDLVNHYJQZWEMMFZESZDDFH65GRA4L5DKA3HEVDZMUA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:30:01.292018959 (177h25m59.84348942s) [noise=true , address=81.171.14.70:4042 , piece_id=L3EX3TO6HZS3PNL5BQRN2NHMEEVBOF23DNESSZZZA6XWJ4OX5F5Q, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:30:02.335554269 (177h26m0.88702473s) [noise=true , address=81.171.14.70:4032 , piece_id=6I6JHCYQGSBKOCFDMWS4D7YLA2BB3X7Q6N4VSYV5WOWIVU7QITWQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:30:03.591420817 (177h26m2.142891318s) [noise=true , address=81.171.14.70:4046 , piece_id=CQRR6T2754TV3FM5U5SDKZKRMMH6A6XUADALCOAGSOXH3G64TB7A, seg_pos={0 0}, upload=156257]: Done uploading piece. err: ecclient: upload failed (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7, address:81.171.14.70:4046): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 17:30:03.591697764 (177h26m2.143168235s) [noise=true , address=81.171.14.70:4027 , piece_id=BA5OGJNDUYVSH5EL7AFRDKMNP44GR7D6Y3BUTFHP6SC2ESWBWY4Q, seg_pos={0 0}, upload=156257]: Done uploading piece. err: ecclient: upload failed (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4, address:81.171.14.70:4027): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 17:30:03.591895902 (177h26m2.143366363s) [noise=true , address=81.171.14.70:4044 , piece_id=GOJ5FCGXG76SD25KRISU23ZW7G6P5ZAKRZLDNWKIFJKBZQVCKDHQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: ecclient: upload failed (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr, address:81.171.14.70:4044): protocol: write tcp 188.166.225.170:60220->81.171.14.70:4044: use of closed network connection; write tcp 188.166.225.170:60220->81.171.14.70:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60220->81.171.14.70:4044: use of closed network connection Nov 16 17:30:03.592099259 (177h26m2.14356972s) [noise=true , address=81.171.14.70:4033 , piece_id=SUI7KRZEHAAOQBXAVEK2Y3J227LSD2APRZNMXODQYT3H6FMB2DXA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: ecclient: upload failed (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8, address:81.171.14.70:4033): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 17:30:03.592137882 (177h26m2.143608344s) [seg_pos={0 0}, upload=156257]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 16 17:30:03.781561870 (177h26m2.333032331s) [upload=156257]: Done uploading segments... Nov 16 17:30:03.781692278 (177h26m2.33316274s) [upload=156257]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.500968943s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.209901147s trace: https://jaeger.dev.storj.io/trace/5a8411d863de3536 FAIL (2025-11-16 17:30:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== YZZQ2NOS2SXOCD76P6XFQUJ64ZRCID7X63CJKOUPQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.293883180 (177h25m2.845353641s) [noise=true , address=81.171.14.70:4044 , piece_id=GOJ5FCGXG76SD25KRISU23ZW7G6P5ZAKRZLDNWKIFJKBZQVCKDHQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.298949508 (177h25m2.850419999s) [noise=true , address=185.167.204.205:4023 , piece_id=XN2QSJ3V2E6IPJNLBEZVHG62A63DRN2ZHIL6YJVSQC2WMY5ELUFQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.299688725 (177h25m2.851159176s) [noise=true , address=89.149.201.147:4002 , piece_id=LKPZVTUHJEBMD4PMCWWQ77XK4EDUINAXTWZUD2CTCZL2MUMZGCHQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.300482637 (177h25m2.851953098s) [noise=true , address=89.149.201.147:4006 , piece_id=HE2XG5ITH2MRJQ6HPKH2TDBKVJY7XWTGVJNNL7K3LNFMMIWEDK2Q, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.301338105 (177h25m2.852808566s) [noise=true , address=81.171.14.70:4042 , piece_id=L3EX3TO6HZS3PNL5BQRN2NHMEEVBOF23DNESSZZZA6XWJ4OX5F5Q, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.301949750 (177h25m2.853420211s) [noise=true , address=89.149.201.147:4001 , piece_id=L5JVJICBH3RTDL5IUKWSDYLD4FYXKIBCUMLBI3ME7SREE3UG6SYQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.302735878 (177h25m2.854206339s) [noise=true , address=81.171.14.70:4033 , piece_id=SUI7KRZEHAAOQBXAVEK2Y3J227LSD2APRZNMXODQYT3H6FMB2DXA, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.304429843 (177h25m2.855900304s) [noise=true , address=81.171.14.70:4030 , piece_id=XQG75D4FPC5BPZUGMVE477FEPONHVBVEJSPQUHPW5QWCW3WECFEQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.306171960 (177h25m2.857642421s) [noise=true , address=185.167.204.205:4005 , piece_id=I7TKD7PUNFYSXEEQUOJATJSZNY7CJKU6VEAMAX26WHXED4GP732A, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.307777438 (177h25m2.859247899s) [noise=true , address=185.167.204.206:4039 , piece_id=PDZNLXGDXOUIAWDS2KTGVEJB5UT2TMBGK5AO4XYKHUEFW2TN2PDQ, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:04.310072839 (177h25m2.8615433s) [noise=true , address=185.167.204.205:4020 , piece_id=ZHEONHQSAGESWXZDUTKJMY4RHLPZUOV4KMZZL46H57XH2PRZG3UA, seg_pos={0 0}, upload=156257]: Uploading piece... Nov 16 17:29:05.660189389 (177h25m4.21165985s) [upload=156257]: Next returned nil segment Nov 16 17:29:05.660208655 (177h25m4.211679126s) [upload=156257]: Waiting for error group managing segments... Nov 16 17:29:06.248075107 (177h25m4.799545568s) [noise=true , address=185.167.204.205:4027 , piece_id=46UM2RN7MTH6E224VUHCGSQ5ENQNRASNRHI642FOQJB2GR4D3PFA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.289496302 (177h25m4.840966763s) [noise=true , address=185.167.204.206:4039 , piece_id=PDZNLXGDXOUIAWDS2KTGVEJB5UT2TMBGK5AO4XYKHUEFW2TN2PDQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.399274525 (177h25m4.950744986s) [noise=true , address=185.167.204.205:4023 , piece_id=XN2QSJ3V2E6IPJNLBEZVHG62A63DRN2ZHIL6YJVSQC2WMY5ELUFQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.435677385 (177h25m4.987147836s) [noise=true , address=185.167.204.206:4069 , piece_id=HPPXP5DQIBTYZZQ2NOS2SXOCD76P6XFQUJ64ZRCID7X63CJKOUPQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.770064945 (177h25m5.321535406s) [noise=true , address=89.149.201.147:4001 , piece_id=L5JVJICBH3RTDL5IUKWSDYLD4FYXKIBCUMLBI3ME7SREE3UG6SYQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.776862789 (177h25m5.32833325s) [noise=true , address=89.149.201.147:4006 , piece_id=HE2XG5ITH2MRJQ6HPKH2TDBKVJY7XWTGVJNNL7K3LNFMMIWEDK2Q, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.782410283 (177h25m5.333880744s) [noise=true , address=89.149.201.147:4005 , piece_id=JVHBZ7N52CN2NBEBQJ3XP3W57PILUBBISMYCAZVDRB55G4JUEKSA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.814083125 (177h25m5.365553586s) [noise=true , address=89.149.201.147:4008 , piece_id=V4JRN3IPIXP2S2SYWV4GB2TWDGHS4PW66DCH2TTQOZLSDJMZJDDQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.868520486 (177h25m5.419990947s) [noise=true , address=185.167.204.206:4059 , piece_id=M5DQ3WJHV7AOSCFCJ442RVDOZP66KB3SP6GHD4JZGOZS7736SYRA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.900967262 (177h25m5.452437723s) [noise=true , address=89.149.201.147:4002 , piece_id=LKPZVTUHJEBMD4PMCWWQ77XK4EDUINAXTWZUD2CTCZL2MUMZGCHQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:06.959077197 (177h25m5.510547658s) [noise=true , address=185.167.204.205:4020 , piece_id=ZHEONHQSAGESWXZDUTKJMY4RHLPZUOV4KMZZL46H57XH2PRZG3UA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:07.043974090 (177h25m5.595444551s) [noise=true , address=185.167.204.205:4005 , piece_id=I7TKD7PUNFYSXEEQUOJATJSZNY7CJKU6VEAMAX26WHXED4GP732A, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:07.317941797 (177h25m5.869412258s) [noise=true , address=89.149.201.147:4014 , piece_id=OAXZ4DISEOHYICZEJAYMZYRTAUTE4OAZGQY26AXNBLAANCIKRU5Q, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:07.941429550 (177h25m6.492900001s) [noise=true , address=185.167.204.206:4064 , piece_id=NGINKCDVHB3FF2IVF3UAYAHCEXCAPOMG52DWAJCSX3I6ROFQE7XQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:08.384861638 (177h25m6.936332099s) [noise=true , address=185.167.204.206:4060 , piece_id=LMLIYFNLFFXITGXS2AC56EIUEURCWZQ4L2WZPAHQ3NRL5QWTMAIA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:09.438981884 (177h25m7.990452345s) [noise=true , address=185.167.204.206:4061 , piece_id=NXJMECD2HAI7DFLCDYRUMI2I5M3WL4U726LJMFRFILX4UJESFXQA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:10.820326839 (177h25m9.3717973s) [noise=true , address=185.167.204.206:4050 , piece_id=XLP3XGNM6GJKARQL4G7AEEIGLV23V5EX7KBZQYXU73WOKZ4R7NIA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:12.299104339 (177h25m10.8505748s) [noise=true , address=185.167.204.205:4004 , piece_id=NQSAX5TNX5SUVGAN5UQ6TCBGW2FJLUEROMTYLGGJWUG3B7ZJDVGA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:28.942804843 (177h25m27.494275304s) [noise=true , address=81.171.14.70:4028 , piece_id=7MVRUM3L57BAYTF6FJUQC5FEHCMXK4ULFLOURXLE5WNLLPYOX7OA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:37.084310989 (177h25m35.63578145s) [noise=true , address=81.171.14.70:4025 , piece_id=23BMULBLFPNYHAMH2YNVMV7SQSJOOWJRMG7PR5LBKW3GCDULQE4A, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:49.339948857 (177h25m47.891419318s) [noise=true , address=81.171.14.70:4030 , piece_id=XQG75D4FPC5BPZUGMVE477FEPONHVBVEJSPQUHPW5QWCW3WECFEQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:29:57.872400335 (177h25m56.423870786s) [noise=true , address=81.171.14.70:4040 , piece_id=IYDSHR5WIHDLVNHYJQZWEMMFZESZDDFH65GRA4L5DKA3HEVDZMUA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:30:01.292018959 (177h25m59.84348942s) [noise=true , address=81.171.14.70:4042 , piece_id=L3EX3TO6HZS3PNL5BQRN2NHMEEVBOF23DNESSZZZA6XWJ4OX5F5Q, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:30:02.335554269 (177h26m0.88702473s) [noise=true , address=81.171.14.70:4032 , piece_id=6I6JHCYQGSBKOCFDMWS4D7YLA2BB3X7Q6N4VSYV5WOWIVU7QITWQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: <nil> Nov 16 17:30:03.591420817 (177h26m2.142891318s) [noise=true , address=81.171.14.70:4046 , piece_id=CQRR6T2754TV3FM5U5SDKZKRMMH6A6XUADALCOAGSOXH3G64TB7A, seg_pos={0 0}, upload=156257]: Done uploading piece. err: ecclient: upload failed (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7, address:81.171.14.70:4046): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 17:30:03.591697764 (177h26m2.143168235s) [noise=true , address=81.171.14.70:4027 , piece_id=BA5OGJNDUYVSH5EL7AFRDKMNP44GR7D6Y3BUTFHP6SC2ESWBWY4Q, seg_pos={0 0}, upload=156257]: Done uploading piece. err: ecclient: upload failed (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4, address:81.171.14.70:4027): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 17:30:03.591895902 (177h26m2.143366363s) [noise=true , address=81.171.14.70:4044 , piece_id=GOJ5FCGXG76SD25KRISU23ZW7G6P5ZAKRZLDNWKIFJKBZQVCKDHQ, seg_pos={0 0}, upload=156257]: Done uploading piece. err: ecclient: upload failed (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr, address:81.171.14.70:4044): protocol: write tcp 188.166.225.170:60220->81.171.14.70:4044: use of closed network connection; write tcp 188.166.225.170:60220->81.171.14.70:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60220->81.171.14.70:4044: use of closed network connection Nov 16 17:30:03.592099259 (177h26m2.14356972s) [noise=true , address=81.171.14.70:4033 , piece_id=SUI7KRZEHAAOQBXAVEK2Y3J227LSD2APRZNMXODQYT3H6FMB2DXA, seg_pos={0 0}, upload=156257]: Done uploading piece. err: ecclient: upload failed (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8, address:81.171.14.70:4033): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 17:30:03.592137882 (177h26m2.143608344s) [seg_pos={0 0}, upload=156257]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 16 17:30:03.781561870 (177h26m2.333032331s) [upload=156257]: Done uploading segments... Nov 16 17:30:03.781692278 (177h26m2.33316274s) [upload=156257]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 1m0.209915043s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.969019982s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:54:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.969118369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.127652746s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:54:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.127672694s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.951405561s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:54:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.951463873s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.051802351s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:53:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.051830476s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.183229805s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:53:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.18343753s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.125831776s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:53:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.12586002s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.181874545s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:53:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.181908799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.171371023s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:52:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.171475312s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.1295845s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:52:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.129651196s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.020866379s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:52:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.020924057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.175288111s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:52:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.175320562s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.373451225s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:51:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.373467175s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.932208537s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:51:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.932245758s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.157429027s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:50:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.157486237s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.951645695s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:50:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.951808895s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.148744138s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:50:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.148782682s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.253177909s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:50:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.253210391s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.080804896s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:49:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.08083867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.926499294s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:49:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.926657456s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.254989228s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:49:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.255017903s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.003538026s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:48:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.003563566s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.106974992s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:47:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.107371285s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.27049766s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:47:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.27052369s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.913775457s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:46:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.913806926s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.113858197s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:46:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.113887443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.939212346s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:46:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.939242944s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.175964662s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:46:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.176001883s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.007450279s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:45:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.007477712s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.08424206s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:45:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.084480675s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.010715634s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:44:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.010747142s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.038718466s trace: https://jaeger.dev.storj.io/trace/3e4560ab56ee4602 FAIL (2025-11-16 16:44:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.03874657s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.009283687s trace: https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e FAIL (2025-11-16 16:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== OHA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments... Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments... Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.009417842s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.977880219s trace: https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a FAIL (2025-11-16 16:43:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece... Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments... Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil> Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments... Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.977909164s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.491654263s trace: https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e FAIL (2025-11-16 16:43:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== OHA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece... Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments... Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil> Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments... Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.491683709s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)