Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.683183963s FAIL (2025-06-28 04:41:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.683206065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.629782301s FAIL (2025-06-28 04:40:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.629809562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.579432795s FAIL (2025-06-28 04:40:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.579464175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.854944632s FAIL (2025-06-28 04:39:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.854975432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.71940323s FAIL (2025-06-28 04:38:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.719422696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.606649695s FAIL (2025-06-28 04:37:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.606677597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.68783784s FAIL (2025-06-28 04:37:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.687869109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.655781017s FAIL (2025-06-28 04:36:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.655806938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.671361571s FAIL (2025-06-28 04:35:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.67139774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.605115838s FAIL (2025-06-28 04:34:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.605144352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.714828607s FAIL (2025-06-28 04:33:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.714855949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.664553794s FAIL (2025-06-28 04:33:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.664582019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.649189418s FAIL (2025-06-28 04:32:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.64921164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.618855585s FAIL (2025-06-28 04:31:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.618877748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.662548429s FAIL (2025-06-28 04:31:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.662586861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.815780023s FAIL (2025-06-28 04:30:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.815808558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.696853919s FAIL (2025-06-28 04:30:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.696878926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.633724281s FAIL (2025-06-28 04:29:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.633756483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.725641982s FAIL (2025-06-28 04:28:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.725671359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.697501981s FAIL (2025-06-28 04:28:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.697536636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.718223949s FAIL (2025-06-28 04:27:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.718240791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.803971421s FAIL (2025-06-28 04:26:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.804021648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.610339091s FAIL (2025-06-28 04:25:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.610367455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.566831195s FAIL (2025-06-28 04:24:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.566852967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.709524032s FAIL (2025-06-28 04:24:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.709552747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.641381786s FAIL (2025-06-28 04:23:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.64141106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.593374078s FAIL (2025-06-28 04:22:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.59340173s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.600404476s FAIL (2025-06-28 04:21:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.60043227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.622125349s FAIL (2025-06-28 04:21:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.622148171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.621410566s FAIL (2025-06-28 04:20:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.621441837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.672273582s FAIL (2025-06-28 04:19:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.672297577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.688938489s FAIL (2025-06-28 04:18:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.688964109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.594955443s FAIL (2025-06-28 04:18:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.594983718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.787781536s FAIL (2025-06-28 04:17:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.787807536s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.666132205s FAIL (2025-06-28 04:16:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.66615539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.78370082s FAIL (2025-06-28 04:15:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.78372593s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.744881593s FAIL (2025-06-28 04:14:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.744909988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.825472169s FAIL (2025-06-28 04:14:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.825501505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.641256624s FAIL (2025-06-28 04:14:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.641290538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.601423459s FAIL (2025-06-28 04:13:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.601443988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.711794144s FAIL (2025-06-28 04:12:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.711817079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.794449346s FAIL (2025-06-28 04:11:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.79451919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.929913995s FAIL (2025-06-28 04:11:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.929941878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.849459932s FAIL (2025-06-28 04:10:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.849486833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.781769248s FAIL (2025-06-28 04:10:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.7817861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.685568143s FAIL (2025-06-28 04:09:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.685594151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.680372892s FAIL (2025-06-28 04:08:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.680417316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 22.433079311s FAIL (2025-06-28 04:07:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 22.433118667s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.244708728s FAIL (2025-06-28 04:06:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 30.24473024s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 10.409876385s target: https://us1.storj.io/login FAIL (2025-06-27 23:03:38 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 10.409908285s core satellite us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)