Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X......X...........................................] duration: 5.821651605s FAIL (2025-09-23 14:32:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.821686411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X......X..........................................] duration: 6.263044477s FAIL (2025-09-23 14:32:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.263148205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...XX......X.................................X..] duration: 7.177938635s FAIL (2025-09-23 14:32:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:30:47.519377614 (59m32.981470097s) [upload=879]: Starting upload Sep 23 14:30:47.519450904 (59m32.981543377s) [upload=879]: Done starting upload Sep 23 14:30:47.519622370 (59m32.981714853s) [upload=879]: Uploading segments... Sep 23 14:30:47.519856386 (59m32.981948869s) [upload=879]: Got next segment. Inline: false Sep 23 14:30:47.519972166 (59m32.982064649s) [upload=879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:30:47.519982527 (59m32.982075s) [upload=879]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:30:47.907384470 (59m33.369476953s) [seg_pos={0 0}, upload=879]: Begin upload segment... Sep 23 14:30:47.907442832 (59m33.369535305s) [seg_pos={0 0}, upload=879]: Done begin upload segment. Sep 23 14:30:47.907468551 (59m33.369561034s) [noise=true , address=45.91.116.10:4082 , piece_id=M2UNRTPS3MA3GFUDWT5LZCAVU2OEBRTU4LDK4JLICR65JI5PM44A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.907501273 (59m33.369593746s) [noise=true , address=145.79.227.19:4079 , piece_id=H2ZXGOQREBCAFDXV4FW6GQMOKBRRMKQDVGYCVDTDF5M3RNL6D4JQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908302589 (59m33.370395092s) [noise=true , address=45.91.116.5:4298 , piece_id=DLAQYV5PNI2MSRRO7XVMMIJ32GUKUZ7TDE5Q3CB5GES2RN7JHCPQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908905226 (59m33.370997709s) [noise=true , address=45.91.116.6:4975 , piece_id=FHB2P3FIO7B5ZZS525I2LQXXB4KDOPXA5XYNWMZHQDIUMVXZX6WQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908915045 (59m33.371007528s) [noise=true , address=145.79.227.16:4909 , piece_id=XESCG3FTTZE4HAWGAKIUC6XTVYUZFTBYFYAE6ZZ7ZBVPBFGHSC5A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.909511421 (59m33.371603894s) [noise=true , address=145.79.227.5:4223 , piece_id=TTDEHY2UNUCWCZYGAKXRR23PI2YO5M3NOTERFKFGIINPYFI356WA, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.909628834 (59m33.371721317s) [noise=true , address=145.79.227.6:4259 , piece_id=COFBXVCWOAZQYOVPLTERLC422JN6YKD6OO3S7HZ5OTIQTDAC6YYQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.910120030 (59m33.372212513s) [noise=true , address=145.79.227.4:4185 , piece_id=WZOOXZYG7OP5B3T7CKUCDHGEQQOH35COQAT6EGL4PLNETXTLGMZQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.910592179 (59m33.372684662s) [noise=true , address=145.79.227.12:4767 , piece_id=VLAIEHPFPVH2NA26HYKORPY4AOECYK5NSNEPR6TZFH5FO37SHG5A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911055531 (59m33.373148014s) [noise=true , address=77.90.31.11:4553 , piece_id=LX72FGFAYXQYJPNT5HE4CV3UXES7TIYNGHPTB34U3UW653SNENLA, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911418722 (59m33.373511205s) [noise=true , address=145.79.227.15:4874 , piece_id=EPRTDN3BZ7ECEED22SAAOX3YWCMEWGACGY33KREZWOV6Z6QDKLCQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911765573 (59m33.373858046s) [noise=true , address=45.91.116.13:4153 , piece_id=2ROXASZOUNQWODFA5LSORBVRL3QKK7TVLMJ2GXAVSB5VQ3QSS5MQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911831599 (59m33.373924072s) [noise=true , address=45.91.116.9:4261 , piece_id=PQPCE2UWB6ZKAUM5HLJT4LD4KVVR5A2BGMRSAKQ3SC3B4LXIURCQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.912323926 (59m33.374416409s) [noise=true , address=45.91.116.11:4119 , piece_id=W3RW4VIQL3NGHZWANJFPATJB5BQGRR7ZS2ORJBPXVRXSZO72JWBQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.912999052 (59m33.375091535s) [noise=true , address=145.79.227.7:4294 , piece_id=IRRMTZS22642CHFXWNRJCUYVQKN2ELFNNKDABCLWRJGJVBSIRXGQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.913617148 (59m33.375709632s) [noise=true , address=145.79.227.17:4969 , piece_id=C7HRWDYQ22JPV2W53E56F6AL47TEUJE5DYEY6IFJWUN54ETFTS3Q, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:50.224478053 (59m35.686570536s) [upload=879]: Next returned nil segment Sep 23 14:30:50.224519211 (59m35.686611684s) [upload=879]: Waiting for error group managing segments... Sep 23 14:30:51.114370090 (59m36.576462563s) [noise=true , address=45.91.116.9:4261 , piece_id=PQPCE2UWB6ZKAUM5HLJT4LD4KVVR5A2BGMRSAKQ3SC3B4LXIURCQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.115043943 (59m36.577136426s) [noise=true , address=45.91.116.10:4082 , piece_id=M2UNRTPS3MA3GFUDWT5LZCAVU2OEBRTU4LDK4JLICR65JI5PM44A, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.158463131 (59m36.620555614s) [noise=true , address=77.90.31.11:4553 , piece_id=LX72FGFAYXQYJPNT5HE4CV3UXES7TIYNGHPTB34U3UW653SNENLA, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.312220734 (59m36.774313217s) [noise=true , address=45.91.116.5:4298 , piece_id=DLAQYV5PNI2MSRRO7XVMMIJ32GUKUZ7TDE5Q3CB5GES2RN7JHCPQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.564121960 (59m37.026214443s) [noise=true , address=45.91.116.11:4119 , piece_id=W3RW4VIQL3NGHZWANJFPATJB5BQGRR7ZS2ORJBPXVRXSZO72JWBQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.618986272 (59m37.081078755s) [noise=true , address=45.91.116.13:4153 , piece_id=2ROXASZOUNQWODFA5LSORBVRL3QKK7TVLMJ2GXAVSB5VQ3QSS5MQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:52.283737671 (59m37.745830144s) [noise=true , address=45.91.116.6:4975 , piece_id=FHB2P3FIO7B5ZZS525I2LQXXB4KDOPXA5XYNWMZHQDIUMVXZX6WQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:56.761671991 (59m42.223764474s) [noise=true , address=145.79.227.12:4767 , piece_id=VLAIEHPFPVH2NA26HYKORPY4AOECYK5NSNEPR6TZFH5FO37SHG5A, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:02.511456285 (59m47.973548768s) [noise=true , address=145.79.227.4:4185 , piece_id=WZOOXZYG7OP5B3T7CKUCDHGEQQOH35COQAT6EGL4PLNETXTLGMZQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:04.580071107 (59m50.04216359s) [noise=true , address=145.79.227.17:4969 , piece_id=C7HRWDYQ22JPV2W53E56F6AL47TEUJE5DYEY6IFJWUN54ETFTS3Q, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:07.593753483 (59m53.055845956s) [noise=true , address=145.79.227.19:4079 , piece_id=H2ZXGOQREBCAFDXV4FW6GQMOKBRRMKQDVGYCVDTDF5M3RNL6D4JQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:16.248941040 (1h0m1.711033523s) [noise=true , address=145.79.227.7:4294 , piece_id=IRRMTZS22642CHFXWNRJCUYVQKN2ELFNNKDABCLWRJGJVBSIRXGQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection Sep 23 14:31:16.249042443 (1h0m1.711134916s) [noise=true , address=145.79.227.16:4909 , piece_id=XESCG3FTTZE4HAWGAKIUC6XTVYUZFTBYFYAE6ZZ7ZBVPBFGHSC5A, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 14:31:16.249119410 (1h0m1.711211883s) [noise=true , address=145.79.227.15:4874 , piece_id=EPRTDN3BZ7ECEED22SAAOX3YWCMEWGACGY33KREZWOV6Z6QDKLCQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection Sep 23 14:31:16.249206846 (1h0m1.711299319s) [noise=true , address=145.79.227.6:4259 , piece_id=COFBXVCWOAZQYOVPLTERLC422JN6YKD6OO3S7HZ5OTIQTDAC6YYQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection Sep 23 14:31:16.249258325 (1h0m1.711350808s) [noise=true , address=145.79.227.5:4223 , piece_id=TTDEHY2UNUCWCZYGAKXRR23PI2YO5M3NOTERFKFGIINPYFI356WA, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 14:31:16.249284624 (1h0m1.711377107s) [seg_pos={0 0}, upload=879]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:31:16.637266714 (1h0m2.099359197s) [upload=879]: Done uploading segments... Sep 23 14:31:16.637306770 (1h0m2.099399253s) [upload=879]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.177974535s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X...XX......X.................................X.] duration: 6.756146666s FAIL (2025-09-23 14:31:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:30:47.519377614 (59m32.981470097s) [upload=879]: Starting upload Sep 23 14:30:47.519450904 (59m32.981543377s) [upload=879]: Done starting upload Sep 23 14:30:47.519622370 (59m32.981714853s) [upload=879]: Uploading segments... Sep 23 14:30:47.519856386 (59m32.981948869s) [upload=879]: Got next segment. Inline: false Sep 23 14:30:47.519972166 (59m32.982064649s) [upload=879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:30:47.519982527 (59m32.982075s) [upload=879]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:30:47.907384470 (59m33.369476953s) [seg_pos={0 0}, upload=879]: Begin upload segment... Sep 23 14:30:47.907442832 (59m33.369535305s) [seg_pos={0 0}, upload=879]: Done begin upload segment. Sep 23 14:30:47.907468551 (59m33.369561034s) [noise=true , address=45.91.116.10:4082 , piece_id=M2UNRTPS3MA3GFUDWT5LZCAVU2OEBRTU4LDK4JLICR65JI5PM44A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.907501273 (59m33.369593746s) [noise=true , address=145.79.227.19:4079 , piece_id=H2ZXGOQREBCAFDXV4FW6GQMOKBRRMKQDVGYCVDTDF5M3RNL6D4JQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908302589 (59m33.370395092s) [noise=true , address=45.91.116.5:4298 , piece_id=DLAQYV5PNI2MSRRO7XVMMIJ32GUKUZ7TDE5Q3CB5GES2RN7JHCPQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908905226 (59m33.370997709s) [noise=true , address=45.91.116.6:4975 , piece_id=FHB2P3FIO7B5ZZS525I2LQXXB4KDOPXA5XYNWMZHQDIUMVXZX6WQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908915045 (59m33.371007528s) [noise=true , address=145.79.227.16:4909 , piece_id=XESCG3FTTZE4HAWGAKIUC6XTVYUZFTBYFYAE6ZZ7ZBVPBFGHSC5A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.909511421 (59m33.371603894s) [noise=true , address=145.79.227.5:4223 , piece_id=TTDEHY2UNUCWCZYGAKXRR23PI2YO5M3NOTERFKFGIINPYFI356WA, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.909628834 (59m33.371721317s) [noise=true , address=145.79.227.6:4259 , piece_id=COFBXVCWOAZQYOVPLTERLC422JN6YKD6OO3S7HZ5OTIQTDAC6YYQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.910120030 (59m33.372212513s) [noise=true , address=145.79.227.4:4185 , piece_id=WZOOXZYG7OP5B3T7CKUCDHGEQQOH35COQAT6EGL4PLNETXTLGMZQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.910592179 (59m33.372684662s) [noise=true , address=145.79.227.12:4767 , piece_id=VLAIEHPFPVH2NA26HYKORPY4AOECYK5NSNEPR6TZFH5FO37SHG5A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911055531 (59m33.373148014s) [noise=true , address=77.90.31.11:4553 , piece_id=LX72FGFAYXQYJPNT5HE4CV3UXES7TIYNGHPTB34U3UW653SNENLA, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911418722 (59m33.373511205s) [noise=true , address=145.79.227.15:4874 , piece_id=EPRTDN3BZ7ECEED22SAAOX3YWCMEWGACGY33KREZWOV6Z6QDKLCQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911765573 (59m33.373858046s) [noise=true , address=45.91.116.13:4153 , piece_id=2ROXASZOUNQWODFA5LSORBVRL3QKK7TVLMJ2GXAVSB5VQ3QSS5MQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911831599 (59m33.373924072s) [noise=true , address=45.91.116.9:4261 , piece_id=PQPCE2UWB6ZKAUM5HLJT4LD4KVVR5A2BGMRSAKQ3SC3B4LXIURCQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.912323926 (59m33.374416409s) [noise=true , address=45.91.116.11:4119 , piece_id=W3RW4VIQL3NGHZWANJFPATJB5BQGRR7ZS2ORJBPXVRXSZO72JWBQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.912999052 (59m33.375091535s) [noise=true , address=145.79.227.7:4294 , piece_id=IRRMTZS22642CHFXWNRJCUYVQKN2ELFNNKDABCLWRJGJVBSIRXGQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.913617148 (59m33.375709632s) [noise=true , address=145.79.227.17:4969 , piece_id=C7HRWDYQ22JPV2W53E56F6AL47TEUJE5DYEY6IFJWUN54ETFTS3Q, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:50.224478053 (59m35.686570536s) [upload=879]: Next returned nil segment Sep 23 14:30:50.224519211 (59m35.686611684s) [upload=879]: Waiting for error group managing segments... Sep 23 14:30:51.114370090 (59m36.576462563s) [noise=true , address=45.91.116.9:4261 , piece_id=PQPCE2UWB6ZKAUM5HLJT4LD4KVVR5A2BGMRSAKQ3SC3B4LXIURCQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.115043943 (59m36.577136426s) [noise=true , address=45.91.116.10:4082 , piece_id=M2UNRTPS3MA3GFUDWT5LZCAVU2OEBRTU4LDK4JLICR65JI5PM44A, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.158463131 (59m36.620555614s) [noise=true , address=77.90.31.11:4553 , piece_id=LX72FGFAYXQYJPNT5HE4CV3UXES7TIYNGHPTB34U3UW653SNENLA, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.312220734 (59m36.774313217s) [noise=true , address=45.91.116.5:4298 , piece_id=DLAQYV5PNI2MSRRO7XVMMIJ32GUKUZ7TDE5Q3CB5GES2RN7JHCPQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.564121960 (59m37.026214443s) [noise=true , address=45.91.116.11:4119 , piece_id=W3RW4VIQL3NGHZWANJFPATJB5BQGRR7ZS2ORJBPXVRXSZO72JWBQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.618986272 (59m37.081078755s) [noise=true , address=45.91.116.13:4153 , piece_id=2ROXASZOUNQWODFA5LSORBVRL3QKK7TVLMJ2GXAVSB5VQ3QSS5MQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:52.283737671 (59m37.745830144s) [noise=true , address=45.91.116.6:4975 , piece_id=FHB2P3FIO7B5ZZS525I2LQXXB4KDOPXA5XYNWMZHQDIUMVXZX6WQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:56.761671991 (59m42.223764474s) [noise=true , address=145.79.227.12:4767 , piece_id=VLAIEHPFPVH2NA26HYKORPY4AOECYK5NSNEPR6TZFH5FO37SHG5A, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:02.511456285 (59m47.973548768s) [noise=true , address=145.79.227.4:4185 , piece_id=WZOOXZYG7OP5B3T7CKUCDHGEQQOH35COQAT6EGL4PLNETXTLGMZQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:04.580071107 (59m50.04216359s) [noise=true , address=145.79.227.17:4969 , piece_id=C7HRWDYQ22JPV2W53E56F6AL47TEUJE5DYEY6IFJWUN54ETFTS3Q, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:07.593753483 (59m53.055845956s) [noise=true , address=145.79.227.19:4079 , piece_id=H2ZXGOQREBCAFDXV4FW6GQMOKBRRMKQDVGYCVDTDF5M3RNL6D4JQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:16.248941040 (1h0m1.711033523s) [noise=true , address=145.79.227.7:4294 , piece_id=IRRMTZS22642CHFXWNRJCUYVQKN2ELFNNKDABCLWRJGJVBSIRXGQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection Sep 23 14:31:16.249042443 (1h0m1.711134916s) [noise=true , address=145.79.227.16:4909 , piece_id=XESCG3FTTZE4HAWGAKIUC6XTVYUZFTBYFYAE6ZZ7ZBVPBFGHSC5A, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 14:31:16.249119410 (1h0m1.711211883s) [noise=true , address=145.79.227.15:4874 , piece_id=EPRTDN3BZ7ECEED22SAAOX3YWCMEWGACGY33KREZWOV6Z6QDKLCQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection Sep 23 14:31:16.249206846 (1h0m1.711299319s) [noise=true , address=145.79.227.6:4259 , piece_id=COFBXVCWOAZQYOVPLTERLC422JN6YKD6OO3S7HZ5OTIQTDAC6YYQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection Sep 23 14:31:16.249258325 (1h0m1.711350808s) [noise=true , address=145.79.227.5:4223 , piece_id=TTDEHY2UNUCWCZYGAKXRR23PI2YO5M3NOTERFKFGIINPYFI356WA, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 14:31:16.249284624 (1h0m1.711377107s) [seg_pos={0 0}, upload=879]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:31:16.637266714 (1h0m2.099359197s) [upload=879]: Done uploading segments... Sep 23 14:31:16.637306770 (1h0m2.099399253s) [upload=879]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.756171152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X...XX......X.................................X] duration: 30.390544058s FAIL (2025-09-23 14:31:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:30:47.519377614 (59m32.981470097s) [upload=879]: Starting upload Sep 23 14:30:47.519450904 (59m32.981543377s) [upload=879]: Done starting upload Sep 23 14:30:47.519622370 (59m32.981714853s) [upload=879]: Uploading segments... Sep 23 14:30:47.519856386 (59m32.981948869s) [upload=879]: Got next segment. Inline: false Sep 23 14:30:47.519972166 (59m32.982064649s) [upload=879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:30:47.519982527 (59m32.982075s) [upload=879]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:30:47.907384470 (59m33.369476953s) [seg_pos={0 0}, upload=879]: Begin upload segment... Sep 23 14:30:47.907442832 (59m33.369535305s) [seg_pos={0 0}, upload=879]: Done begin upload segment. Sep 23 14:30:47.907468551 (59m33.369561034s) [noise=true , address=45.91.116.10:4082 , piece_id=M2UNRTPS3MA3GFUDWT5LZCAVU2OEBRTU4LDK4JLICR65JI5PM44A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.907501273 (59m33.369593746s) [noise=true , address=145.79.227.19:4079 , piece_id=H2ZXGOQREBCAFDXV4FW6GQMOKBRRMKQDVGYCVDTDF5M3RNL6D4JQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908302589 (59m33.370395092s) [noise=true , address=45.91.116.5:4298 , piece_id=DLAQYV5PNI2MSRRO7XVMMIJ32GUKUZ7TDE5Q3CB5GES2RN7JHCPQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908905226 (59m33.370997709s) [noise=true , address=45.91.116.6:4975 , piece_id=FHB2P3FIO7B5ZZS525I2LQXXB4KDOPXA5XYNWMZHQDIUMVXZX6WQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.908915045 (59m33.371007528s) [noise=true , address=145.79.227.16:4909 , piece_id=XESCG3FTTZE4HAWGAKIUC6XTVYUZFTBYFYAE6ZZ7ZBVPBFGHSC5A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.909511421 (59m33.371603894s) [noise=true , address=145.79.227.5:4223 , piece_id=TTDEHY2UNUCWCZYGAKXRR23PI2YO5M3NOTERFKFGIINPYFI356WA, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.909628834 (59m33.371721317s) [noise=true , address=145.79.227.6:4259 , piece_id=COFBXVCWOAZQYOVPLTERLC422JN6YKD6OO3S7HZ5OTIQTDAC6YYQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.910120030 (59m33.372212513s) [noise=true , address=145.79.227.4:4185 , piece_id=WZOOXZYG7OP5B3T7CKUCDHGEQQOH35COQAT6EGL4PLNETXTLGMZQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.910592179 (59m33.372684662s) [noise=true , address=145.79.227.12:4767 , piece_id=VLAIEHPFPVH2NA26HYKORPY4AOECYK5NSNEPR6TZFH5FO37SHG5A, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911055531 (59m33.373148014s) [noise=true , address=77.90.31.11:4553 , piece_id=LX72FGFAYXQYJPNT5HE4CV3UXES7TIYNGHPTB34U3UW653SNENLA, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911418722 (59m33.373511205s) [noise=true , address=145.79.227.15:4874 , piece_id=EPRTDN3BZ7ECEED22SAAOX3YWCMEWGACGY33KREZWOV6Z6QDKLCQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911765573 (59m33.373858046s) [noise=true , address=45.91.116.13:4153 , piece_id=2ROXASZOUNQWODFA5LSORBVRL3QKK7TVLMJ2GXAVSB5VQ3QSS5MQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.911831599 (59m33.373924072s) [noise=true , address=45.91.116.9:4261 , piece_id=PQPCE2UWB6ZKAUM5HLJT4LD4KVVR5A2BGMRSAKQ3SC3B4LXIURCQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.912323926 (59m33.374416409s) [noise=true , address=45.91.116.11:4119 , piece_id=W3RW4VIQL3NGHZWANJFPATJB5BQGRR7ZS2ORJBPXVRXSZO72JWBQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.912999052 (59m33.375091535s) [noise=true , address=145.79.227.7:4294 , piece_id=IRRMTZS22642CHFXWNRJCUYVQKN2ELFNNKDABCLWRJGJVBSIRXGQ, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:47.913617148 (59m33.375709632s) [noise=true , address=145.79.227.17:4969 , piece_id=C7HRWDYQ22JPV2W53E56F6AL47TEUJE5DYEY6IFJWUN54ETFTS3Q, seg_pos={0 0}, upload=879]: Uploading piece... Sep 23 14:30:50.224478053 (59m35.686570536s) [upload=879]: Next returned nil segment Sep 23 14:30:50.224519211 (59m35.686611684s) [upload=879]: Waiting for error group managing segments... Sep 23 14:30:51.114370090 (59m36.576462563s) [noise=true , address=45.91.116.9:4261 , piece_id=PQPCE2UWB6ZKAUM5HLJT4LD4KVVR5A2BGMRSAKQ3SC3B4LXIURCQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.115043943 (59m36.577136426s) [noise=true , address=45.91.116.10:4082 , piece_id=M2UNRTPS3MA3GFUDWT5LZCAVU2OEBRTU4LDK4JLICR65JI5PM44A, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.158463131 (59m36.620555614s) [noise=true , address=77.90.31.11:4553 , piece_id=LX72FGFAYXQYJPNT5HE4CV3UXES7TIYNGHPTB34U3UW653SNENLA, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.312220734 (59m36.774313217s) [noise=true , address=45.91.116.5:4298 , piece_id=DLAQYV5PNI2MSRRO7XVMMIJ32GUKUZ7TDE5Q3CB5GES2RN7JHCPQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.564121960 (59m37.026214443s) [noise=true , address=45.91.116.11:4119 , piece_id=W3RW4VIQL3NGHZWANJFPATJB5BQGRR7ZS2ORJBPXVRXSZO72JWBQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:51.618986272 (59m37.081078755s) [noise=true , address=45.91.116.13:4153 , piece_id=2ROXASZOUNQWODFA5LSORBVRL3QKK7TVLMJ2GXAVSB5VQ3QSS5MQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:52.283737671 (59m37.745830144s) [noise=true , address=45.91.116.6:4975 , piece_id=FHB2P3FIO7B5ZZS525I2LQXXB4KDOPXA5XYNWMZHQDIUMVXZX6WQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:30:56.761671991 (59m42.223764474s) [noise=true , address=145.79.227.12:4767 , piece_id=VLAIEHPFPVH2NA26HYKORPY4AOECYK5NSNEPR6TZFH5FO37SHG5A, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:02.511456285 (59m47.973548768s) [noise=true , address=145.79.227.4:4185 , piece_id=WZOOXZYG7OP5B3T7CKUCDHGEQQOH35COQAT6EGL4PLNETXTLGMZQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:04.580071107 (59m50.04216359s) [noise=true , address=145.79.227.17:4969 , piece_id=C7HRWDYQ22JPV2W53E56F6AL47TEUJE5DYEY6IFJWUN54ETFTS3Q, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:07.593753483 (59m53.055845956s) [noise=true , address=145.79.227.19:4079 , piece_id=H2ZXGOQREBCAFDXV4FW6GQMOKBRRMKQDVGYCVDTDF5M3RNL6D4JQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: <nil> Sep 23 14:31:16.248941040 (1h0m1.711033523s) [noise=true , address=145.79.227.7:4294 , piece_id=IRRMTZS22642CHFXWNRJCUYVQKN2ELFNNKDABCLWRJGJVBSIRXGQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60786->145.79.227.7:4294: use of closed network connection Sep 23 14:31:16.249042443 (1h0m1.711134916s) [noise=true , address=145.79.227.16:4909 , piece_id=XESCG3FTTZE4HAWGAKIUC6XTVYUZFTBYFYAE6ZZ7ZBVPBFGHSC5A, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 14:31:16.249119410 (1h0m1.711211883s) [noise=true , address=145.79.227.15:4874 , piece_id=EPRTDN3BZ7ECEED22SAAOX3YWCMEWGACGY33KREZWOV6Z6QDKLCQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33828->145.79.227.15:4874: use of closed network connection Sep 23 14:31:16.249206846 (1h0m1.711299319s) [noise=true , address=145.79.227.6:4259 , piece_id=COFBXVCWOAZQYOVPLTERLC422JN6YKD6OO3S7HZ5OTIQTDAC6YYQ, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54632->145.79.227.6:4259: use of closed network connection Sep 23 14:31:16.249258325 (1h0m1.711350808s) [noise=true , address=145.79.227.5:4223 , piece_id=TTDEHY2UNUCWCZYGAKXRR23PI2YO5M3NOTERFKFGIINPYFI356WA, seg_pos={0 0}, upload=879]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 14:31:16.249284624 (1h0m1.711377107s) [seg_pos={0 0}, upload=879]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:31:16.637266714 (1h0m2.099359197s) [upload=879]: Done uploading segments... Sep 23 14:31:16.637306770 (1h0m2.099399253s) [upload=879]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.390573594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X......X.........................................] duration: 6.601109566s FAIL (2025-09-23 14:31:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.601134793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X......X........................................] duration: 39.702455971s FAIL (2025-09-23 14:31:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 39.70249848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X...XX......X.................................] duration: 6.975352974s FAIL (2025-09-23 14:30:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.975398499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X......X.......................................] duration: 6.343158652s FAIL (2025-09-23 14:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.343261558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X...XX......X................................] duration: 6.730462239s FAIL (2025-09-23 14:29:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.730494652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X......X......................................] duration: 6.346563504s FAIL (2025-09-23 14:29:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.346616745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X...XX......X...............................] duration: 7.232854464s FAIL (2025-09-23 14:29:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.232890091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X......X.....................................] duration: 6.159670454s FAIL (2025-09-23 14:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.159701594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X...XX......X..............................] duration: 9.675183087s FAIL (2025-09-23 14:28:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.675212603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X......X....................................] duration: 6.080704663s FAIL (2025-09-23 14:27:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.08073505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X...XX......X.............................] duration: 6.492608073s FAIL (2025-09-23 14:27:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.492639512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X......X...................................] duration: 5.938434791s FAIL (2025-09-23 14:26:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.938469828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X...XX......X............................] duration: 6.263041933s FAIL (2025-09-23 14:26:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.26307647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X......X..................................] duration: 5.872176453s FAIL (2025-09-23 14:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.872207172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X...XX......X...........................] duration: 8.08725417s FAIL (2025-09-23 14:25:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.087284467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X......X.................................] duration: 5.845716152s FAIL (2025-09-23 14:25:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.84577313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X...XX......X..........................] duration: 6.452437007s FAIL (2025-09-23 14:24:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.452472093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X......X................................] duration: 5.806794703s FAIL (2025-09-23 14:24:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.806828497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X......X...............................] duration: 6.207431351s FAIL (2025-09-23 14:24:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.207465517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X...XX......X.........................] duration: 7.248000927s FAIL (2025-09-23 14:23:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.248036825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X...XX......X........................] duration: 6.970321443s FAIL (2025-09-23 14:23:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.970353936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X......X..............................] duration: 7.169057145s FAIL (2025-09-23 14:23:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 7.169091821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X...XX......X.......................] duration: 8.561836782s FAIL (2025-09-23 14:22:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.561870497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......X.............................] duration: 6.063983701s FAIL (2025-09-23 14:22:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.064019089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X...XX......X......................] duration: 14.016893601s FAIL (2025-09-23 14:22:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 14.016928737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X...XX......X.....................] duration: 7.80780261s FAIL (2025-09-23 14:21:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.807835874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X......X............................] duration: 6.092108897s FAIL (2025-09-23 14:21:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.092143493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X...XX......X....................] duration: 8.452372781s FAIL (2025-09-23 14:20:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.452424229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......X...........................] duration: 6.190707917s FAIL (2025-09-23 14:20:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.190744417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X...XX......X...................] duration: 7.137693995s FAIL (2025-09-23 14:20:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.137732107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......X..........................] duration: 6.186774778s FAIL (2025-09-23 14:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.186849219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X...XX......X..................] duration: 6.850238313s FAIL (2025-09-23 14:19:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.850271656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......X.........................] duration: 8.085832269s FAIL (2025-09-23 14:18:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 8.085867796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......X........................] duration: 7.808366365s FAIL (2025-09-23 14:18:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 7.808398867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X...XX......X.................] duration: 8.532775178s FAIL (2025-09-23 14:17:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.53281798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......X.......................] duration: 5.966734869s FAIL (2025-09-23 14:17:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.966769445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X...XX......X................] duration: 7.315781983s FAIL (2025-09-23 14:17:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.315816328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......X......................] duration: 6.024928451s FAIL (2025-09-23 14:16:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 6.02496463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...............................X...XX......X...............] duration: 6.522194244s FAIL (2025-09-23 14:16:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.522225343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......X.....................] duration: 5.983558658s FAIL (2025-09-23 14:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.983592913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...............................X...XX......X..............] duration: 7.179905899s FAIL (2025-09-23 14:16:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.179944022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X......X....................] duration: 5.951379699s FAIL (2025-09-23 14:15:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.951415928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...............................X...XX......X.............] duration: 6.960431016s FAIL (2025-09-23 14:15:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.960465433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......X...................] duration: 5.815045943s FAIL (2025-09-23 14:14:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, Piece ID: IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ) context deadline exceeded; piecestore: (Node ID: 1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, Piece ID: MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA) context deadline exceeded; piecestore: (Node ID: 12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, Piece ID: HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A) context deadline exceeded; piecestore: (Node ID: 1N5XGFCAo5JX9Wmgsgt2iarMQW3A6Q5rkdEEyBVx6beWDeC7WQ, Piece ID: X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA) context deadline exceeded ======LOGS====== 23 14:01:48.436230883 (30m33.898323366s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436426535 (30m33.898519018s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.436939602 (30m33.899032076s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437349453 (30m33.899441936s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.437760115 (30m33.899852608s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438176368 (30m33.900268851s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438208618 (30m33.900301092s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438618339 (30m33.900710812s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.438635943 (30m33.900728426s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439030664 (30m33.901123147s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439102341 (30m33.901194824s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439560032 (30m33.901652515s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.439971676 (30m33.902064159s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440389211 (30m33.902481694s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440805243 (30m33.902897736s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.440981829 (30m33.903074313s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:48.441428360 (30m33.903520843s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Uploading piece... Sep 23 14:01:50.031218324 (30m35.493310807s) [upload=453]: Next returned nil segment Sep 23 14:01:50.031337330 (30m35.493429843s) [upload=453]: Waiting for error group managing segments... Sep 23 14:01:50.448811348 (30m35.910903831s) [noise=true , address=45.91.116.12:4241 , piece_id=PTMGKKCDZHPRCKLWKNL374OX4I2QNBYM7JQ7DHZ5Z3ME7MANK3XQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.490166886 (30m35.952259369s) [noise=true , address=77.90.31.18:4824 , piece_id=VIOZ7GBI2VW4IJLOTGQ26JSCTL7YIVANIQ26JXSD4XLAMX76TMJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.511592294 (30m35.973684777s) [noise=true , address=45.91.116.11:4150 , piece_id=X53SMXMFAMMN46LR3ZRBZEZFZMANUIO7J7OB6RKGPY3UVCDZP4GA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:50.570009361 (30m36.032101844s) [noise=true , address=77.90.31.11:4587 , piece_id=YNZSJU34ZS5NYHCLROXNYSXTGWJSDLPNBBPTLDS5DENZC3OA6YVA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.250641401 (30m36.712733884s) [noise=true , address=145.79.227.21:4478 , piece_id=IJWYAKYGC2D72IPAMOGUWZWKSJ5JR3I2TTHPRRM2Q35KXNYZIEIQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.267698241 (30m36.729790724s) [noise=true , address=145.79.227.11:4757 , piece_id=P4TOYQRV5M3ESYMDSA6H2IS5O4OUUGKPKZED3IGBTC2LGJ62SIFA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.333279319 (30m36.795371802s) [noise=true , address=145.79.227.14:4860 , piece_id=R773DSUAU5GH7FBFCYZW7E7MT2LHZPBETYHQMXDQFYXOZW4VHZTA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.789773138 (30m37.251865621s) [noise=true , address=145.79.227.28:4740 , piece_id=MURUBPW7AC3OCLNQVIUPR23AZCBR73OL2NSVJ4YVZYR34LT2QZGA, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:51.855236342 (30m37.317328825s) [noise=true , address=145.79.227.41:4108 , piece_id=MFP5WJRWXQFTCVO52UZI6D4VJNEBFTRSKPLEWU6TPDLMF7CWKJ7Q, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.189709461 (30m37.651801944s) [noise=true , address=145.79.227.19:4088 , piece_id=HAOHLF2UP5LOZ3X3FFGQ3KH4XJTFHJXAWUEH4NCJR4XGRNWGUS5A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.257019562 (30m37.719112045s) [noise=true , address=145.79.227.15:4889 , piece_id=CJFKYSAQGADYVYXR2XMQQD7PQPAWUFRN2AXW745RWRVQ5JEWIDKQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.719495212 (30m38.181587695s) [noise=true , address=145.79.227.40:4053 , piece_id=ANHJMI6M47QCFUSY75TUPC7TY2A47UVVIWDYBOQTU7LVF2NQ4Z2A, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:52.774265735 (30m38.236358218s) [noise=true , address=145.79.227.16:4913 , piece_id=IWXBUDLOIM772MDNYU26HMEEURT7DT4D3GVSAOI3M6G3ES4VONHQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113101661 (30m38.575194144s) [noise=true , address=145.79.227.12:4791 , piece_id=4IH4KTE2HCVRW3PYCXEUSDQMB3LIRTARYTOVU63RINGUDKF2AEJQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: <nil> Sep 23 14:01:53.113139974 (30m38.575232447s) [seg_pos={0 0}, upload=453]: Segment reached optimal threshold of 14 pieces. Sep 23 14:01:53.113454262 (30m38.575546736s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49): protocol: expected piece hash; context canceled; EOF Sep 23 14:01:53.113464142 (30m38.575556615s) [noise=true , address=145.79.227.23:4578 , piece_id=P24XBFTCYFJ6SLHJSHWEMHAMGBE6OSNP6N67232M3VSD7BMMBYDA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113696985 (30m38.575789468s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp): protocol: read tcp 188.166.225.170:57472->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113705532 (30m38.575798005s) [noise=true , address=145.79.227.18:4010 , piece_id=76RNOMEZPYCRERBBPJH2ZAZQZZ2XLU2ROS6ZGX2LSYY6HUMH4UFQ, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.113869473 (30m38.575961956s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp): protocol: read tcp 188.166.225.170:47358->145.79.227.25:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 14:01:53.113881437 (30m38.57597391s) [noise=true , address=145.79.227.25:4628 , piece_id=352I5DXPS7NIW7HOYK22OWQIFMO3G4SBSW5METBPFGZVV2IEN65A, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114010151 (30m38.576102624s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: read tcp 188.166.225.170:53300->145.79.227.49:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Sep 23 14:01:53.114019660 (30m38.576112133s) [noise=true , address=145.79.227.49:4352 , piece_id=7NJNXGMMXDKB3HDC4UUK6QNSGZHBWJXQH7QCV4ABDV5QKPNUAQNA, seg_pos={0 0}, upload=453]: Long tail cancellation due to optimal threshold Sep 23 14:01:53.114039567 (30m38.57613205s) [seg_pos={0 0}, upload=453]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 23 14:01:53.114125471 (30m38.576217944s) [upload=453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 14:01:53.114131663 (30m38.576224136s) [upload=453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 14:01:53.564871204 (30m39.026963687s) [upload=453]: Done uploading segments... Sep 23 14:01:53.564896622 (30m39.026989095s) [upload=453]: Upload finished. err: <nil> Duration: 5.815091049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...............................X...XX......X............] duration: 6.069538121s FAIL (2025-09-23 14:13:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 14:05:13.975659754 (33m59.437752227s) [upload=503]: Starting upload Sep 23 14:05:13.975762019 (33m59.437854512s) [upload=503]: Done starting upload Sep 23 14:05:13.975820901 (33m59.437913384s) [upload=503]: Uploading segments... Sep 23 14:05:13.976008598 (33m59.438101081s) [upload=503]: Got next segment. Inline: false Sep 23 14:05:13.976062752 (33m59.438155235s) [upload=503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:05:13.976068743 (33m59.438161226s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:05:14.228748772 (33m59.690841245s) [seg_pos={0 0}, upload=503]: Begin upload segment... Sep 23 14:05:14.228804819 (33m59.690897292s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Sep 23 14:05:14.228828734 (33m59.690921217s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.229958115 (33m59.692050588s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230296830 (33m59.692389313s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.230683156 (33m59.692775639s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231013464 (33m59.693105937s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231307264 (33m59.693399747s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.231652000 (33m59.693744473s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.232888445 (33m59.694980928s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.234277460 (33m59.696369943s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.235672726 (33m59.697765209s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.237439140 (33m59.699531623s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.238604889 (33m59.700697373s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240221058 (33m59.702313541s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.240750525 (33m59.702842999s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242427469 (33m59.704519952s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:14.242560512 (33m59.704652995s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Uploading piece... Sep 23 14:05:16.058395838 (34m1.520488321s) [upload=503]: Next returned nil segment Sep 23 14:05:16.058420345 (34m1.520512828s) [upload=503]: Waiting for error group managing segments... Sep 23 14:05:16.351844008 (34m1.813936491s) [noise=true , address=77.90.31.16:4735 , piece_id=M2LK53XVM4V7NUQ6VBXKKIA6I7AMCDUAULBG7MYTZENYTN6ULSRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.442982243 (34m1.905074726s) [noise=true , address=45.91.116.6:4975 , piece_id=JFHT4CTJ6UCU5BEFDVGO7YWKPNMWHBFUYNYHL7P4AJACCBKNVR3Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.448229624 (34m1.910322107s) [noise=true , address=77.90.31.21:4877 , piece_id=KAAPGOHQZHJRZXBYWPNSV4A645C6R2YXVL6TU2CEHLJZR3IO2TRA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.449108999 (34m1.911201482s) [noise=true , address=45.91.116.9:4261 , piece_id=OHTEQAWPPC5CI6AL2SSAPB6SOX3CBJBP5Q5ZLBSDOBIWYOIIAQUA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.452605497 (34m1.91469798s) [noise=true , address=45.91.116.7:4011 , piece_id=FYG3UYYCEMOAJMWENI6LZ32XBXBPB56GYG3POUVH67XZFF3KQD4Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.467530627 (34m1.92962311s) [noise=true , address=45.91.116.4:4938 , piece_id=6SP5BDUEQCDLMNUHLMGNL4OZJNKPJRYLGXHWWNZBDNDVHFJITKSQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.483123248 (34m1.945215731s) [noise=true , address=77.90.31.19:4843 , piece_id=U7KUHVSTUII2ECSSEPZFBXOBET4QGWYQY7DVQA3TAZHYSILEJJWQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.515465092 (34m1.977557576s) [noise=true , address=45.91.116.8:4047 , piece_id=TNDGA4TTOMCAYHUZKOLLKE4S5GZ7FUSFFUAW5W5XM5TA7APGHXNA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:16.581541625 (34m2.043634109s) [noise=true , address=77.90.31.18:4806 , piece_id=WPHCMKQ5XQSSHDNMVTDXGV62NW7NZ4WZDE6SHGA3CWNJTSCFKZLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.197125783 (34m19.659218256s) [noise=true , address=145.79.227.5:4222 , piece_id=53HITJVBCHUKYCRELISGG7ARRUBLYAOLYUGS2EVKNSLCZXIM4FNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:34.701382550 (34m20.163475023s) [noise=true , address=145.79.227.2:4113 , piece_id=BMJFE62MUKUSJRCCPZEKBKPIX772Z53KYW2QOMLOONGMEMGJX7QA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Sep 23 14:05:43.290842238 (34m28.752934741s) [noise=true , address=145.79.227.18:4007 , piece_id=MKBBVTW7E34TYD2DPYQEHQ7Z4D7VVLMOF6TG4H3BAAQRFYDE3W3A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.18:4007: use of closed network connection Sep 23 14:05:43.290869840 (34m28.752962353s) [noise=true , address=145.79.227.11:4729 , piece_id=XDMCGHEMTOO7J3JYEY7AMSLSGQRZRM4DKLGCYALTMMVNYTOXL3MQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40850->145.79.227.11:4729: use of closed network connection Sep 23 14:05:43.290938952 (34m28.753031435s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6NSUTRLHBLSY2OGGAEMMP3YPN3HR6MSVYEKWBMRVILPMUSQPRIQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56510->145.79.227.4:4186: use of closed network connection Sep 23 14:05:43.290970191 (34m28.753062684s) [noise=true , address=145.79.227.12:4767 , piece_id=NE37WOQ4QGQQHCKXEJUDLFABG46ASTL7L4MKJ7YIDAZYF3X7JPGA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46790->145.79.227.12:4767: use of closed network connection Sep 23 14:05:43.291006039 (34m28.753098513s) [noise=true , address=145.79.227.19:4079 , piece_id=NPK5XNQ6HNAME33IWVAAWTO7BWW3Y7747GSLKIBEKHFPBFCGSOKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46230->145.79.227.19:4079: use of closed network connection Sep 23 14:05:43.291034744 (34m28.753127227s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 14:05:43.561153501 (34m29.023245985s) [upload=503]: Done uploading segments... Sep 23 14:05:43.561188288 (34m29.023280771s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.069570443s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)