Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.017803265s FAIL (2025-07-16 19:00:08 UTC) orbiter warning: context deadline exceeded Duration: 6.017828462s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.83248065s FAIL (2025-07-16 18:59:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832497883s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.279198382s FAIL (2025-07-16 18:59:38 UTC) orbiter warning: context deadline exceeded Duration: 6.279219472s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.837267956s FAIL (2025-07-16 18:59:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.837295801s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.464226421s FAIL (2025-07-16 18:59:08 UTC) orbiter warning: context deadline exceeded Duration: 6.464270446s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.772657167s FAIL (2025-07-16 18:59:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.772683169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.713060693s FAIL (2025-07-16 18:58:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.713085531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.728809271s FAIL (2025-07-16 18:58:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.728838917s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.081143626s FAIL (2025-07-16 18:57:56 UTC) orbiter warning: context deadline exceeded Duration: 6.081162963s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.800155199s FAIL (2025-07-16 18:57:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.80018141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.668369893s FAIL (2025-07-16 18:57:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.66839435s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.200282575s FAIL (2025-07-16 18:57:07 UTC) orbiter warning: context deadline exceeded Duration: 6.200304707s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.219781391s FAIL (2025-07-16 18:56:36 UTC) orbiter warning: context deadline exceeded Duration: 6.219801959s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.737127691s FAIL (2025-07-16 18:56:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.737147908s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.308025828s FAIL (2025-07-16 18:55:39 UTC) orbiter warning: context deadline exceeded Duration: 6.308051506s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.910233246s FAIL (2025-07-16 18:55:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.910257373s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.83365907s FAIL (2025-07-16 18:55:02 UTC) orbiter warning: context deadline exceeded Duration: 5.833705188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.829562861s FAIL (2025-07-16 18:54:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.829589132s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.871292184s FAIL (2025-07-16 18:54:41 UTC) orbiter warning: context deadline exceeded Duration: 5.871319016s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.821502601s FAIL (2025-07-16 18:54:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.821527507s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.269475305s FAIL (2025-07-16 18:53:40 UTC) orbiter warning: context deadline exceeded Duration: 6.269499021s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.017795665s FAIL (2025-07-16 18:53:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.017822938s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.024612913s FAIL (2025-07-16 18:52:59 UTC) orbiter warning: context deadline exceeded Duration: 6.024633864s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.954073625s FAIL (2025-07-16 18:52:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.954099545s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.198596005s FAIL (2025-07-16 18:52:26 UTC) orbiter warning: context deadline exceeded Duration: 6.198619459s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.764627481s FAIL (2025-07-16 18:51:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.764659432s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.961382582s FAIL (2025-07-16 18:51:22 UTC) orbiter warning: context deadline exceeded Duration: 5.961422589s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.80923032s FAIL (2025-07-16 18:50:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.809253685s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.295923708s FAIL (2025-07-16 18:50:42 UTC) orbiter warning: context deadline exceeded Duration: 6.295943827s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.86450035s FAIL (2025-07-16 18:50:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.86452111s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.339824229s FAIL (2025-07-16 18:50:12 UTC) orbiter warning: context deadline exceeded Duration: 6.339993133s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.824994142s FAIL (2025-07-16 18:49:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.825022076s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.055876803s FAIL (2025-07-16 18:49:04 UTC) orbiter warning: context deadline exceeded Duration: 6.055896379s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.811870073s FAIL (2025-07-16 18:48:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.811906013s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.324177353s FAIL (2025-07-16 18:48:37 UTC) orbiter warning: context deadline exceeded Duration: 6.324188645s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00097793s FAIL (2025-07-16 18:48:30 UTC) orbiter warning: context deadline exceeded Duration: 30.000998429s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.022212773s FAIL (2025-07-16 18:47:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.022238131s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.859744919s FAIL (2025-07-16 18:46:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.85977195s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.740342604s FAIL (2025-07-16 18:45:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.740374294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.705302348s FAIL (2025-07-16 18:44:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.705328437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.789799481s FAIL (2025-07-16 18:43:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.789818859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.729114069s FAIL (2025-07-16 18:43:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.729137293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.849274457s FAIL (2025-07-16 18:42:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.84929712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.795380501s FAIL (2025-07-16 18:41:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.795403305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.236128257s FAIL (2025-07-16 18:40:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== equestItem_ObjectCommit Jul 16 18:40:14.162346704 (561h29m48.932988039s) [upload=448998]: Done uploading segments... Jul 16 18:40:14.162364248 (561h29m48.933005583s) [upload=448998]: Upload finished. err: <nil> Jul 16 18:40:26.790061078 (561h30m1.560702403s) [upload=449004]: Starting upload Jul 16 18:40:26.790122086 (561h30m1.560763421s) [upload=449004]: Done starting upload Jul 16 18:40:26.790255701 (561h30m1.560897046s) [upload=449004]: Uploading segments... Jul 16 18:40:26.790358027 (561h30m1.560999362s) [upload=449004]: Got next segment. Inline: false Jul 16 18:40:26.790393765 (561h30m1.5610351s) [upload=449004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 18:40:26.790399216 (561h30m1.561040551s) [upload=449004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 18:40:27.024250342 (561h30m1.794891677s) [seg_pos={0 0}, upload=449004]: Begin upload segment... Jul 16 18:40:27.024289247 (561h30m1.794930572s) [seg_pos={0 0}, upload=449004]: Done begin upload segment. Jul 16 18:40:27.024309335 (561h30m1.79495066s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024354372 (561h30m1.794995707s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024647202 (561h30m1.795288537s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024829300 (561h30m1.795470635s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.024893494 (561h30m1.795534829s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025182626 (561h30m1.795823961s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025429118 (561h30m1.796070443s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.025669789 (561h30m1.796311124s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026478125 (561h30m1.79711946s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.026742371 (561h30m1.797383706s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.027023589 (561h30m1.797664924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.028864902 (561h30m1.799506237s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029216585 (561h30m1.79985791s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029521789 (561h30m1.800163124s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.029930290 (561h30m1.800571625s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:27.030254411 (561h30m1.800895746s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Uploading piece... Jul 16 18:40:28.767225210 (561h30m3.537866535s) [upload=449004]: Next returned nil segment Jul 16 18:40:28.767306556 (561h30m3.537947891s) [upload=449004]: Waiting for error group managing segments... Jul 16 18:40:29.085631721 (561h30m3.856273056s) [noise=true , address=45.91.116.4:4938 , piece_id=MH57FQJW7LSKPCVNLOYPCGV35CL6GRLIZUT27SI5Y3NTKNOS4KIQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.092662945 (561h30m3.86330428s) [noise=true , address=45.91.116.5:4297 , piece_id=LDOJ366N3RNGF7YBVLDWFH4VKG43QV2VCB74JSYYSZQFP3PGZGCA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.131839973 (561h30m3.902481318s) [noise=true , address=45.91.116.13:4155 , piece_id=SLYG2S6VQUTBDAABI47YPXL6UCDHD3SRIVVDEMLAUGLR63CLFSRQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.146887965 (561h30m3.9175293s) [noise=true , address=104.36.82.8:4969 , piece_id=SVZWKGIUM6MABUDFUR3CN5ZMHLE7GVHMRBF6SFEOJSLJOTSINK4A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.162142703 (561h30m3.932784038s) [noise=true , address=104.36.82.14:4221 , piece_id=ILPLYEJ45YVE3QVHRJBZAKN7EGFZTMASHOKFYG7Z74R6AVTIIN4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.169925444 (561h30m3.940566779s) [noise=true , address=104.36.82.12:4151 , piece_id=Z76CBNLX5Z42KX6Q6HOKBQ4W2ZYO7VF622LSO6QPBA7YUZN5YNVA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.288603616 (561h30m4.059244951s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMEQDTY2BN3KFHDXOHCS3WSQN2PJI5JJJ2NYZ75ISPFR6GHX2XZA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.304769627 (561h30m4.075410962s) [noise=true , address=45.91.116.9:4263 , piece_id=ZNOASUR47WVW6BXURSBTLHYSC42KEHXOJJLPSV3WRR2NESZ7OV7Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.359375276 (561h30m4.130016611s) [noise=true , address=45.91.116.12:4225 , piece_id=ZUIY35X44GRSOOFWSJZJLOGWSWTIF4TGZU4XG5CWEL7426WIM6BQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.371795961 (561h30m4.142437296s) [noise=true , address=104.36.82.16:4294 , piece_id=EBLMBQFJ37QXGJHMFTPGXGWRAMYXIZJVMHRJ6FAEY6EO5GS636GA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:29.571621466 (561h30m4.342262791s) [noise=true , address=45.91.116.10:4081 , piece_id=FEN5O7E65LQYVBV2DEL2PQZ7NJCFGVJ3HGWFIMDR5ZNQT6PCSDAA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:51.206870849 (561h30m25.977512204s) [noise=true , address=77.90.31.13:4627 , piece_id=XYH2VDMXJZFT2JGRZ6TMK64QJ5HDRBQADDBHLKVGCQ62HPI4I4WA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: <nil> Jul 16 18:40:56.145115448 (561h30m30.915756783s) [noise=true , address=77.90.31.10:4517 , piece_id=RA2EVKJIC2KXXBCHHO222B7R4ZXY2DLVZ273ZOZNZHSWCRE3KCWA, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41382->77.90.31.10:4517: use of closed network connection Jul 16 18:40:56.145372068 (561h30m30.916013404s) [noise=true , address=77.90.31.21:4879 , piece_id=I24V4OGTG27JFZOF36JN66IWEFUHN35TGFLEOCQBRPR3HDKNGTUQ, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45536->77.90.31.21:4879: use of closed network connection Jul 16 18:40:56.145410391 (561h30m30.916051726s) [noise=true , address=77.90.31.16:4735 , piece_id=4BZZLOFJQOOEDQOVIDWV4UCVJCSEWCONAEQD2YTJWWA7QTZCJM4Q, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57378->77.90.31.16:4735: use of closed network connection Jul 16 18:40:56.145436632 (561h30m30.916077967s) [noise=true , address=77.90.31.16:4734 , piece_id=KFKECQTO2RM5XY6SIYHIRDV27QMTN6BSQAPXUIXKJFX7JDY5CQ5A, seg_pos={0 0}, upload=449004]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58498->77.90.31.16:4734: use of closed network connection Jul 16 18:40:56.145457642 (561h30m30.916098977s) [seg_pos={0 0}, upload=449004]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 18:40:56.379144706 (561h30m31.149786041s) [upload=449004]: Done uploading segments... Jul 16 18:40:56.379169884 (561h30m31.149811219s) [upload=449004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.236144527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.773223786s FAIL (2025-07-16 18:18:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.773274182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.862760791s FAIL (2025-07-16 18:17:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.862778094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.805251878s FAIL (2025-07-16 18:16:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.805273579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.769223622s FAIL (2025-07-16 18:15:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.769255603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.812032079s FAIL (2025-07-16 18:14:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.81205828s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)