Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.268502947s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:56:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 7.268531322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 15.373218446s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:56:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 15.373243904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X..X.X....X.....X.....X......X...X..........] duration: 7.442111577s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:55:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 7.442142565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 12.033480705s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:55:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 12.033505202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 15.352558184s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:55:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 15.352581019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 44.513511432s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:54:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 44.513541631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X..X.X....X.....X.....X......X...X.........] duration: 5.876490697s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:54:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 5.876511116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X..X.X....X.....X.....X......X...X........] duration: 6.236633891s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 6.236699074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X..X.X....X.....X.....X......X...X.......] duration: 16.982604747s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:53:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 16.982649883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 18.70547185s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:53:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 18.705502158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X..X.X....X.....X.....X......X...X......] duration: 5.973436734s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:53:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 5.973467423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X..X.X....X.....X.....X......X...X.....] duration: 10.821941103s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:52:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 10.822086541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.678949218s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 6.678987881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 11.235596252s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:51:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 11.235622382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X..X.X....X.....X.....X......X...X....] duration: 10.47580545s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:51:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 10.475832511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X..X.X....X.....X.....X......X...X...] duration: 5.913578686s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:50:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 5.913601389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 10.719744347s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:50:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 10.719766448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 15.385748045s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:50:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 15.385779935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 36.725422164s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:50:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 36.725439787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X..X.X....X.....X.....X......X...X..] duration: 5.97384893s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:50:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 5.973881631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 30.111632253s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:49:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 30.111664635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................X..X.X....X.....X.....X......X...X.] duration: 12.402666582s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:49:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 12.402691219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 15.606724193s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 15.606752245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................X..X.X....X.....X.....X......X...X] duration: 30.002638975s trace: https://jaeger.dev.storj.io/trace/156ad5a012a53700 FAIL (2025-12-28 23:48:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .189688552 (560h19m2.060771587s) [upload=491232]: Starting upload Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments... Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment... Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment. Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece... Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments... Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566 , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230 , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303 , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730 , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015 , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978 , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910 , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877 , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083 , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119 , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046 , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298 , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974 , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil> Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces. Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086 , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195 , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294 , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments... Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil> Duration: 30.002659885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 18.914502755s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:48:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 18.914528875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 15.436242508s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:47:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 15.436334122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X..X.X....X.....X.....X......X...] duration: 5.895653008s trace: https://jaeger.dev.storj.io/trace/116d16f16aa65029 FAIL (2025-12-28 23:47:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA) context deadline exceeded ======LOGS====== ts... Dec 28 23:43:36.830130260 (560h14m36.701213305s) [upload=491167]: Got next segment. Inline: false Dec 28 23:43:36.830568504 (560h14m36.701651539s) [upload=491167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:43:36.830574456 (560h14m36.701657491s) [upload=491167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:43:37.075146784 (560h14m36.946229849s) [seg_pos={0 0}, upload=491167]: Begin upload segment... Dec 28 23:43:37.075242266 (560h14m36.946325311s) [seg_pos={0 0}, upload=491167]: Done begin upload segment. Dec 28 23:43:37.075336636 (560h14m36.946419691s) [noise=true , address=145.79.227.11:4731 , piece_id=AHZPNTM7F52TGSQDTSTWRGOUZABVTTCLYRLH4WVMV4XINTXKC2GA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.076249454 (560h14m36.947332499s) [noise=true , address=77.90.31.13:4627 , piece_id=C55E2RX3CYTPDK4OQZVZ7CL5Q2H2UUFE2GWBIWQ3YHS7YX4Z4Q4A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.076443334 (560h14m36.947526369s) [noise=true , address=145.79.227.18:4006 , piece_id=7MRVO7F57MGJSEMR5DGVQUUESRPZISJU35ACCPM2SGWVYXIWRC2A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077153395 (560h14m36.94823643s) [noise=true , address=45.91.116.6:4974 , piece_id=Z5CDZ7V5J26OBXQADTXPL3DP5BW7OJWZONSNBBAYQMM23BCCVP2A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077181860 (560h14m36.948264905s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077700437 (560h14m36.948783482s) [noise=true , address=145.79.227.41:4087 , piece_id=SUS2IAYQ373OTZ6HJWHQCA5UQNEX3KAGTPZ2EGK5M2W3OZXPCS6Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077783074 (560h14m36.948866119s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078104397 (560h14m36.949187442s) [noise=true , address=145.79.227.4:4186 , piece_id=I3NAINPUDXSDH7NV2QN2UB6NX4EN2GP6KLSHCDRUGGSZBEHVHH4Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078487035 (560h14m36.949570081s) [noise=true , address=145.79.227.24:4582 , piece_id=X4AC4PQZY5NBI6H5OJNO6VUHYD5RB7CMG5ITLXJTZ6453IY42H4Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078700011 (560h14m36.949783046s) [noise=true , address=45.91.116.7:4009 , piece_id=TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078886806 (560h14m36.949969851s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079311274 (560h14m36.950394309s) [noise=true , address=45.91.116.9:4261 , piece_id=XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079342704 (560h14m36.950425739s) [noise=true , address=145.79.227.51:4410 , piece_id=4HJIXHBBUNMN3VJKBMBXU7JHJUAS3BPA2CXGEMVUTKRBMT3SK6IQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079915195 (560h14m36.95099825s) [noise=true , address=145.79.227.15:4874 , piece_id=L6BS46IBD5ITXQ7MGZNEIENIDZF5ALYP2CGU57SU64AHNZABPLSA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.080524174 (560h14m36.951607219s) [noise=true , address=145.79.227.29:4942 , piece_id=7PUEOTIEV4NLXK2FSZ7NMQ4GLN4MULCMOQECMRED3ZPU7ILVZZBQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.081180664 (560h14m36.952263739s) [noise=true , address=145.79.227.39:4015 , piece_id=BJIKZACRR7UEY5TF6OH4MDTZBBYZH5WBYS3TYL6VAVRYJZWQ4GXA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:38.794292952 (560h14m38.665375987s) [upload=491167]: Next returned nil segment Dec 28 23:43:38.794307099 (560h14m38.665390134s) [upload=491167]: Waiting for error group managing segments... Dec 28 23:43:39.069952189 (560h14m38.941035224s) [noise=true , address=145.79.227.15:4874 , piece_id=L6BS46IBD5ITXQ7MGZNEIENIDZF5ALYP2CGU57SU64AHNZABPLSA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.082958227 (560h14m38.954041272s) [noise=true , address=145.79.227.11:4731 , piece_id=AHZPNTM7F52TGSQDTSTWRGOUZABVTTCLYRLH4WVMV4XINTXKC2GA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.088880856 (560h14m38.959963901s) [noise=true , address=145.79.227.41:4087 , piece_id=SUS2IAYQ373OTZ6HJWHQCA5UQNEX3KAGTPZ2EGK5M2W3OZXPCS6Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.089473584 (560h14m38.960556619s) [noise=true , address=145.79.227.51:4410 , piece_id=4HJIXHBBUNMN3VJKBMBXU7JHJUAS3BPA2CXGEMVUTKRBMT3SK6IQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.097208684 (560h14m38.968291719s) [noise=true , address=145.79.227.4:4186 , piece_id=I3NAINPUDXSDH7NV2QN2UB6NX4EN2GP6KLSHCDRUGGSZBEHVHH4Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.103007416 (560h14m38.974090461s) [noise=true , address=145.79.227.39:4015 , piece_id=BJIKZACRR7UEY5TF6OH4MDTZBBYZH5WBYS3TYL6VAVRYJZWQ4GXA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.105721265 (560h14m38.97680431s) [noise=true , address=145.79.227.18:4006 , piece_id=7MRVO7F57MGJSEMR5DGVQUUESRPZISJU35ACCPM2SGWVYXIWRC2A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.120504326 (560h14m38.991587371s) [noise=true , address=145.79.227.24:4582 , piece_id=X4AC4PQZY5NBI6H5OJNO6VUHYD5RB7CMG5ITLXJTZ6453IY42H4Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.255764241 (560h14m39.126847287s) [noise=true , address=77.90.31.13:4627 , piece_id=C55E2RX3CYTPDK4OQZVZ7CL5Q2H2UUFE2GWBIWQ3YHS7YX4Z4Q4A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.358734818 (560h14m39.229817853s) [noise=true , address=45.91.116.6:4974 , piece_id=Z5CDZ7V5J26OBXQADTXPL3DP5BW7OJWZONSNBBAYQMM23BCCVP2A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.364635204 (560h14m39.235718239s) [noise=true , address=45.91.116.9:4261 , piece_id=XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.376457377 (560h14m39.247540412s) [noise=true , address=145.79.227.29:4942 , piece_id=7PUEOTIEV4NLXK2FSZ7NMQ4GLN4MULCMOQECMRED3ZPU7ILVZZBQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.454946674 (560h14m39.326029719s) [noise=true , address=45.91.116.7:4009 , piece_id=TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.454969588 (560h14m39.326052633s) [seg_pos={0 0}, upload=491167]: Segment reached optimal threshold of 13 pieces. Dec 28 23:43:39.455212381 (560h14m39.326295416s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection Dec 28 23:43:39.455221698 (560h14m39.326304743s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455310047 (560h14m39.326393092s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection Dec 28 23:43:39.455319585 (560h14m39.32640263s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455587786 (560h14m39.326670821s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:43:39.455614255 (560h14m39.32669729s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455638041 (560h14m39.326721076s) [seg_pos={0 0}, upload=491167]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:43:39.455759512 (560h14m39.326842557s) [upload=491167]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:43:39.455765264 (560h14m39.326848299s) [upload=491167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:43:39.750546135 (560h14m39.62162917s) [upload=491167]: Done uploading segments... Dec 28 23:43:39.750565742 (560h14m39.621648777s) [upload=491167]: Upload finished. err: <nil> Duration: 5.895681764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 28.352989669s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:46:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 28.353020599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X..X.X....X.....X.....X......X..] duration: 6.2831098s trace: https://jaeger.dev.storj.io/trace/116d16f16aa65029 FAIL (2025-12-28 23:45:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA) context deadline exceeded ======LOGS====== ts... Dec 28 23:43:36.830130260 (560h14m36.701213305s) [upload=491167]: Got next segment. Inline: false Dec 28 23:43:36.830568504 (560h14m36.701651539s) [upload=491167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:43:36.830574456 (560h14m36.701657491s) [upload=491167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:43:37.075146784 (560h14m36.946229849s) [seg_pos={0 0}, upload=491167]: Begin upload segment... Dec 28 23:43:37.075242266 (560h14m36.946325311s) [seg_pos={0 0}, upload=491167]: Done begin upload segment. Dec 28 23:43:37.075336636 (560h14m36.946419691s) [noise=true , address=145.79.227.11:4731 , piece_id=AHZPNTM7F52TGSQDTSTWRGOUZABVTTCLYRLH4WVMV4XINTXKC2GA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.076249454 (560h14m36.947332499s) [noise=true , address=77.90.31.13:4627 , piece_id=C55E2RX3CYTPDK4OQZVZ7CL5Q2H2UUFE2GWBIWQ3YHS7YX4Z4Q4A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.076443334 (560h14m36.947526369s) [noise=true , address=145.79.227.18:4006 , piece_id=7MRVO7F57MGJSEMR5DGVQUUESRPZISJU35ACCPM2SGWVYXIWRC2A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077153395 (560h14m36.94823643s) [noise=true , address=45.91.116.6:4974 , piece_id=Z5CDZ7V5J26OBXQADTXPL3DP5BW7OJWZONSNBBAYQMM23BCCVP2A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077181860 (560h14m36.948264905s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077700437 (560h14m36.948783482s) [noise=true , address=145.79.227.41:4087 , piece_id=SUS2IAYQ373OTZ6HJWHQCA5UQNEX3KAGTPZ2EGK5M2W3OZXPCS6Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077783074 (560h14m36.948866119s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078104397 (560h14m36.949187442s) [noise=true , address=145.79.227.4:4186 , piece_id=I3NAINPUDXSDH7NV2QN2UB6NX4EN2GP6KLSHCDRUGGSZBEHVHH4Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078487035 (560h14m36.949570081s) [noise=true , address=145.79.227.24:4582 , piece_id=X4AC4PQZY5NBI6H5OJNO6VUHYD5RB7CMG5ITLXJTZ6453IY42H4Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078700011 (560h14m36.949783046s) [noise=true , address=45.91.116.7:4009 , piece_id=TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078886806 (560h14m36.949969851s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079311274 (560h14m36.950394309s) [noise=true , address=45.91.116.9:4261 , piece_id=XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079342704 (560h14m36.950425739s) [noise=true , address=145.79.227.51:4410 , piece_id=4HJIXHBBUNMN3VJKBMBXU7JHJUAS3BPA2CXGEMVUTKRBMT3SK6IQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079915195 (560h14m36.95099825s) [noise=true , address=145.79.227.15:4874 , piece_id=L6BS46IBD5ITXQ7MGZNEIENIDZF5ALYP2CGU57SU64AHNZABPLSA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.080524174 (560h14m36.951607219s) [noise=true , address=145.79.227.29:4942 , piece_id=7PUEOTIEV4NLXK2FSZ7NMQ4GLN4MULCMOQECMRED3ZPU7ILVZZBQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.081180664 (560h14m36.952263739s) [noise=true , address=145.79.227.39:4015 , piece_id=BJIKZACRR7UEY5TF6OH4MDTZBBYZH5WBYS3TYL6VAVRYJZWQ4GXA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:38.794292952 (560h14m38.665375987s) [upload=491167]: Next returned nil segment Dec 28 23:43:38.794307099 (560h14m38.665390134s) [upload=491167]: Waiting for error group managing segments... Dec 28 23:43:39.069952189 (560h14m38.941035224s) [noise=true , address=145.79.227.15:4874 , piece_id=L6BS46IBD5ITXQ7MGZNEIENIDZF5ALYP2CGU57SU64AHNZABPLSA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.082958227 (560h14m38.954041272s) [noise=true , address=145.79.227.11:4731 , piece_id=AHZPNTM7F52TGSQDTSTWRGOUZABVTTCLYRLH4WVMV4XINTXKC2GA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.088880856 (560h14m38.959963901s) [noise=true , address=145.79.227.41:4087 , piece_id=SUS2IAYQ373OTZ6HJWHQCA5UQNEX3KAGTPZ2EGK5M2W3OZXPCS6Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.089473584 (560h14m38.960556619s) [noise=true , address=145.79.227.51:4410 , piece_id=4HJIXHBBUNMN3VJKBMBXU7JHJUAS3BPA2CXGEMVUTKRBMT3SK6IQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.097208684 (560h14m38.968291719s) [noise=true , address=145.79.227.4:4186 , piece_id=I3NAINPUDXSDH7NV2QN2UB6NX4EN2GP6KLSHCDRUGGSZBEHVHH4Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.103007416 (560h14m38.974090461s) [noise=true , address=145.79.227.39:4015 , piece_id=BJIKZACRR7UEY5TF6OH4MDTZBBYZH5WBYS3TYL6VAVRYJZWQ4GXA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.105721265 (560h14m38.97680431s) [noise=true , address=145.79.227.18:4006 , piece_id=7MRVO7F57MGJSEMR5DGVQUUESRPZISJU35ACCPM2SGWVYXIWRC2A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.120504326 (560h14m38.991587371s) [noise=true , address=145.79.227.24:4582 , piece_id=X4AC4PQZY5NBI6H5OJNO6VUHYD5RB7CMG5ITLXJTZ6453IY42H4Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.255764241 (560h14m39.126847287s) [noise=true , address=77.90.31.13:4627 , piece_id=C55E2RX3CYTPDK4OQZVZ7CL5Q2H2UUFE2GWBIWQ3YHS7YX4Z4Q4A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.358734818 (560h14m39.229817853s) [noise=true , address=45.91.116.6:4974 , piece_id=Z5CDZ7V5J26OBXQADTXPL3DP5BW7OJWZONSNBBAYQMM23BCCVP2A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.364635204 (560h14m39.235718239s) [noise=true , address=45.91.116.9:4261 , piece_id=XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.376457377 (560h14m39.247540412s) [noise=true , address=145.79.227.29:4942 , piece_id=7PUEOTIEV4NLXK2FSZ7NMQ4GLN4MULCMOQECMRED3ZPU7ILVZZBQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.454946674 (560h14m39.326029719s) [noise=true , address=45.91.116.7:4009 , piece_id=TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.454969588 (560h14m39.326052633s) [seg_pos={0 0}, upload=491167]: Segment reached optimal threshold of 13 pieces. Dec 28 23:43:39.455212381 (560h14m39.326295416s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection Dec 28 23:43:39.455221698 (560h14m39.326304743s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455310047 (560h14m39.326393092s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection Dec 28 23:43:39.455319585 (560h14m39.32640263s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455587786 (560h14m39.326670821s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:43:39.455614255 (560h14m39.32669729s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455638041 (560h14m39.326721076s) [seg_pos={0 0}, upload=491167]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:43:39.455759512 (560h14m39.326842557s) [upload=491167]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:43:39.455765264 (560h14m39.326848299s) [upload=491167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:43:39.750546135 (560h14m39.62162917s) [upload=491167]: Done uploading segments... Dec 28 23:43:39.750565742 (560h14m39.621648777s) [upload=491167]: Upload finished. err: <nil> Duration: 6.283155566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 16.77204492s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:45:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 16.772083833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 18.575011168s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:45:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 18.575078064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................X..X.X....X.....X.....X......X.] duration: 7.440054611s trace: https://jaeger.dev.storj.io/trace/116d16f16aa65029 FAIL (2025-12-28 23:44:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA) context deadline exceeded ======LOGS====== ts... Dec 28 23:43:36.830130260 (560h14m36.701213305s) [upload=491167]: Got next segment. Inline: false Dec 28 23:43:36.830568504 (560h14m36.701651539s) [upload=491167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:43:36.830574456 (560h14m36.701657491s) [upload=491167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:43:37.075146784 (560h14m36.946229849s) [seg_pos={0 0}, upload=491167]: Begin upload segment... Dec 28 23:43:37.075242266 (560h14m36.946325311s) [seg_pos={0 0}, upload=491167]: Done begin upload segment. Dec 28 23:43:37.075336636 (560h14m36.946419691s) [noise=true , address=145.79.227.11:4731 , piece_id=AHZPNTM7F52TGSQDTSTWRGOUZABVTTCLYRLH4WVMV4XINTXKC2GA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.076249454 (560h14m36.947332499s) [noise=true , address=77.90.31.13:4627 , piece_id=C55E2RX3CYTPDK4OQZVZ7CL5Q2H2UUFE2GWBIWQ3YHS7YX4Z4Q4A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.076443334 (560h14m36.947526369s) [noise=true , address=145.79.227.18:4006 , piece_id=7MRVO7F57MGJSEMR5DGVQUUESRPZISJU35ACCPM2SGWVYXIWRC2A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077153395 (560h14m36.94823643s) [noise=true , address=45.91.116.6:4974 , piece_id=Z5CDZ7V5J26OBXQADTXPL3DP5BW7OJWZONSNBBAYQMM23BCCVP2A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077181860 (560h14m36.948264905s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077700437 (560h14m36.948783482s) [noise=true , address=145.79.227.41:4087 , piece_id=SUS2IAYQ373OTZ6HJWHQCA5UQNEX3KAGTPZ2EGK5M2W3OZXPCS6Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077783074 (560h14m36.948866119s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078104397 (560h14m36.949187442s) [noise=true , address=145.79.227.4:4186 , piece_id=I3NAINPUDXSDH7NV2QN2UB6NX4EN2GP6KLSHCDRUGGSZBEHVHH4Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078487035 (560h14m36.949570081s) [noise=true , address=145.79.227.24:4582 , piece_id=X4AC4PQZY5NBI6H5OJNO6VUHYD5RB7CMG5ITLXJTZ6453IY42H4Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078700011 (560h14m36.949783046s) [noise=true , address=45.91.116.7:4009 , piece_id=TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078886806 (560h14m36.949969851s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079311274 (560h14m36.950394309s) [noise=true , address=45.91.116.9:4261 , piece_id=XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079342704 (560h14m36.950425739s) [noise=true , address=145.79.227.51:4410 , piece_id=4HJIXHBBUNMN3VJKBMBXU7JHJUAS3BPA2CXGEMVUTKRBMT3SK6IQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079915195 (560h14m36.95099825s) [noise=true , address=145.79.227.15:4874 , piece_id=L6BS46IBD5ITXQ7MGZNEIENIDZF5ALYP2CGU57SU64AHNZABPLSA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.080524174 (560h14m36.951607219s) [noise=true , address=145.79.227.29:4942 , piece_id=7PUEOTIEV4NLXK2FSZ7NMQ4GLN4MULCMOQECMRED3ZPU7ILVZZBQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.081180664 (560h14m36.952263739s) [noise=true , address=145.79.227.39:4015 , piece_id=BJIKZACRR7UEY5TF6OH4MDTZBBYZH5WBYS3TYL6VAVRYJZWQ4GXA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:38.794292952 (560h14m38.665375987s) [upload=491167]: Next returned nil segment Dec 28 23:43:38.794307099 (560h14m38.665390134s) [upload=491167]: Waiting for error group managing segments... Dec 28 23:43:39.069952189 (560h14m38.941035224s) [noise=true , address=145.79.227.15:4874 , piece_id=L6BS46IBD5ITXQ7MGZNEIENIDZF5ALYP2CGU57SU64AHNZABPLSA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.082958227 (560h14m38.954041272s) [noise=true , address=145.79.227.11:4731 , piece_id=AHZPNTM7F52TGSQDTSTWRGOUZABVTTCLYRLH4WVMV4XINTXKC2GA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.088880856 (560h14m38.959963901s) [noise=true , address=145.79.227.41:4087 , piece_id=SUS2IAYQ373OTZ6HJWHQCA5UQNEX3KAGTPZ2EGK5M2W3OZXPCS6Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.089473584 (560h14m38.960556619s) [noise=true , address=145.79.227.51:4410 , piece_id=4HJIXHBBUNMN3VJKBMBXU7JHJUAS3BPA2CXGEMVUTKRBMT3SK6IQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.097208684 (560h14m38.968291719s) [noise=true , address=145.79.227.4:4186 , piece_id=I3NAINPUDXSDH7NV2QN2UB6NX4EN2GP6KLSHCDRUGGSZBEHVHH4Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.103007416 (560h14m38.974090461s) [noise=true , address=145.79.227.39:4015 , piece_id=BJIKZACRR7UEY5TF6OH4MDTZBBYZH5WBYS3TYL6VAVRYJZWQ4GXA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.105721265 (560h14m38.97680431s) [noise=true , address=145.79.227.18:4006 , piece_id=7MRVO7F57MGJSEMR5DGVQUUESRPZISJU35ACCPM2SGWVYXIWRC2A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.120504326 (560h14m38.991587371s) [noise=true , address=145.79.227.24:4582 , piece_id=X4AC4PQZY5NBI6H5OJNO6VUHYD5RB7CMG5ITLXJTZ6453IY42H4Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.255764241 (560h14m39.126847287s) [noise=true , address=77.90.31.13:4627 , piece_id=C55E2RX3CYTPDK4OQZVZ7CL5Q2H2UUFE2GWBIWQ3YHS7YX4Z4Q4A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.358734818 (560h14m39.229817853s) [noise=true , address=45.91.116.6:4974 , piece_id=Z5CDZ7V5J26OBXQADTXPL3DP5BW7OJWZONSNBBAYQMM23BCCVP2A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.364635204 (560h14m39.235718239s) [noise=true , address=45.91.116.9:4261 , piece_id=XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.376457377 (560h14m39.247540412s) [noise=true , address=145.79.227.29:4942 , piece_id=7PUEOTIEV4NLXK2FSZ7NMQ4GLN4MULCMOQECMRED3ZPU7ILVZZBQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.454946674 (560h14m39.326029719s) [noise=true , address=45.91.116.7:4009 , piece_id=TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.454969588 (560h14m39.326052633s) [seg_pos={0 0}, upload=491167]: Segment reached optimal threshold of 13 pieces. Dec 28 23:43:39.455212381 (560h14m39.326295416s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection Dec 28 23:43:39.455221698 (560h14m39.326304743s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455310047 (560h14m39.326393092s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection Dec 28 23:43:39.455319585 (560h14m39.32640263s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455587786 (560h14m39.326670821s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:43:39.455614255 (560h14m39.32669729s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455638041 (560h14m39.326721076s) [seg_pos={0 0}, upload=491167]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:43:39.455759512 (560h14m39.326842557s) [upload=491167]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:43:39.455765264 (560h14m39.326848299s) [upload=491167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:43:39.750546135 (560h14m39.62162917s) [upload=491167]: Done uploading segments... Dec 28 23:43:39.750565742 (560h14m39.621648777s) [upload=491167]: Upload finished. err: <nil> Duration: 7.440078186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 30.082051117s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:44:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 30.082083098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................X..X.X....X.....X.....X......X] duration: 30.002998089s trace: https://jaeger.dev.storj.io/trace/116d16f16aa65029 FAIL (2025-12-28 23:44:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA) context deadline exceeded ======LOGS====== ts... Dec 28 23:43:36.830130260 (560h14m36.701213305s) [upload=491167]: Got next segment. Inline: false Dec 28 23:43:36.830568504 (560h14m36.701651539s) [upload=491167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:43:36.830574456 (560h14m36.701657491s) [upload=491167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:43:37.075146784 (560h14m36.946229849s) [seg_pos={0 0}, upload=491167]: Begin upload segment... Dec 28 23:43:37.075242266 (560h14m36.946325311s) [seg_pos={0 0}, upload=491167]: Done begin upload segment. Dec 28 23:43:37.075336636 (560h14m36.946419691s) [noise=true , address=145.79.227.11:4731 , piece_id=AHZPNTM7F52TGSQDTSTWRGOUZABVTTCLYRLH4WVMV4XINTXKC2GA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.076249454 (560h14m36.947332499s) [noise=true , address=77.90.31.13:4627 , piece_id=C55E2RX3CYTPDK4OQZVZ7CL5Q2H2UUFE2GWBIWQ3YHS7YX4Z4Q4A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.076443334 (560h14m36.947526369s) [noise=true , address=145.79.227.18:4006 , piece_id=7MRVO7F57MGJSEMR5DGVQUUESRPZISJU35ACCPM2SGWVYXIWRC2A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077153395 (560h14m36.94823643s) [noise=true , address=45.91.116.6:4974 , piece_id=Z5CDZ7V5J26OBXQADTXPL3DP5BW7OJWZONSNBBAYQMM23BCCVP2A, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077181860 (560h14m36.948264905s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077700437 (560h14m36.948783482s) [noise=true , address=145.79.227.41:4087 , piece_id=SUS2IAYQ373OTZ6HJWHQCA5UQNEX3KAGTPZ2EGK5M2W3OZXPCS6Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.077783074 (560h14m36.948866119s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078104397 (560h14m36.949187442s) [noise=true , address=145.79.227.4:4186 , piece_id=I3NAINPUDXSDH7NV2QN2UB6NX4EN2GP6KLSHCDRUGGSZBEHVHH4Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078487035 (560h14m36.949570081s) [noise=true , address=145.79.227.24:4582 , piece_id=X4AC4PQZY5NBI6H5OJNO6VUHYD5RB7CMG5ITLXJTZ6453IY42H4Q, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078700011 (560h14m36.949783046s) [noise=true , address=45.91.116.7:4009 , piece_id=TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.078886806 (560h14m36.949969851s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079311274 (560h14m36.950394309s) [noise=true , address=45.91.116.9:4261 , piece_id=XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079342704 (560h14m36.950425739s) [noise=true , address=145.79.227.51:4410 , piece_id=4HJIXHBBUNMN3VJKBMBXU7JHJUAS3BPA2CXGEMVUTKRBMT3SK6IQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.079915195 (560h14m36.95099825s) [noise=true , address=145.79.227.15:4874 , piece_id=L6BS46IBD5ITXQ7MGZNEIENIDZF5ALYP2CGU57SU64AHNZABPLSA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.080524174 (560h14m36.951607219s) [noise=true , address=145.79.227.29:4942 , piece_id=7PUEOTIEV4NLXK2FSZ7NMQ4GLN4MULCMOQECMRED3ZPU7ILVZZBQ, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:37.081180664 (560h14m36.952263739s) [noise=true , address=145.79.227.39:4015 , piece_id=BJIKZACRR7UEY5TF6OH4MDTZBBYZH5WBYS3TYL6VAVRYJZWQ4GXA, seg_pos={0 0}, upload=491167]: Uploading piece... Dec 28 23:43:38.794292952 (560h14m38.665375987s) [upload=491167]: Next returned nil segment Dec 28 23:43:38.794307099 (560h14m38.665390134s) [upload=491167]: Waiting for error group managing segments... Dec 28 23:43:39.069952189 (560h14m38.941035224s) [noise=true , address=145.79.227.15:4874 , piece_id=L6BS46IBD5ITXQ7MGZNEIENIDZF5ALYP2CGU57SU64AHNZABPLSA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.082958227 (560h14m38.954041272s) [noise=true , address=145.79.227.11:4731 , piece_id=AHZPNTM7F52TGSQDTSTWRGOUZABVTTCLYRLH4WVMV4XINTXKC2GA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.088880856 (560h14m38.959963901s) [noise=true , address=145.79.227.41:4087 , piece_id=SUS2IAYQ373OTZ6HJWHQCA5UQNEX3KAGTPZ2EGK5M2W3OZXPCS6Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.089473584 (560h14m38.960556619s) [noise=true , address=145.79.227.51:4410 , piece_id=4HJIXHBBUNMN3VJKBMBXU7JHJUAS3BPA2CXGEMVUTKRBMT3SK6IQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.097208684 (560h14m38.968291719s) [noise=true , address=145.79.227.4:4186 , piece_id=I3NAINPUDXSDH7NV2QN2UB6NX4EN2GP6KLSHCDRUGGSZBEHVHH4Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.103007416 (560h14m38.974090461s) [noise=true , address=145.79.227.39:4015 , piece_id=BJIKZACRR7UEY5TF6OH4MDTZBBYZH5WBYS3TYL6VAVRYJZWQ4GXA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.105721265 (560h14m38.97680431s) [noise=true , address=145.79.227.18:4006 , piece_id=7MRVO7F57MGJSEMR5DGVQUUESRPZISJU35ACCPM2SGWVYXIWRC2A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.120504326 (560h14m38.991587371s) [noise=true , address=145.79.227.24:4582 , piece_id=X4AC4PQZY5NBI6H5OJNO6VUHYD5RB7CMG5ITLXJTZ6453IY42H4Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.255764241 (560h14m39.126847287s) [noise=true , address=77.90.31.13:4627 , piece_id=C55E2RX3CYTPDK4OQZVZ7CL5Q2H2UUFE2GWBIWQ3YHS7YX4Z4Q4A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.358734818 (560h14m39.229817853s) [noise=true , address=45.91.116.6:4974 , piece_id=Z5CDZ7V5J26OBXQADTXPL3DP5BW7OJWZONSNBBAYQMM23BCCVP2A, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.364635204 (560h14m39.235718239s) [noise=true , address=45.91.116.9:4261 , piece_id=XT4BWH3FNSOXMBWJC3CGBEIGGCMAD5QOOCXODFZFEN63OJJU2JHA, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.376457377 (560h14m39.247540412s) [noise=true , address=145.79.227.29:4942 , piece_id=7PUEOTIEV4NLXK2FSZ7NMQ4GLN4MULCMOQECMRED3ZPU7ILVZZBQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.454946674 (560h14m39.326029719s) [noise=true , address=45.91.116.7:4009 , piece_id=TJ7JVNYM5GWEDWZYUAPLQFXJLU3P6YUIKXVFQVDREKSF45L7MTIQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: <nil> Dec 28 23:43:39.454969588 (560h14m39.326052633s) [seg_pos={0 0}, upload=491167]: Segment reached optimal threshold of 13 pieces. Dec 28 23:43:39.455212381 (560h14m39.326295416s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36634->45.91.116.10:4084: use of closed network connection Dec 28 23:43:39.455221698 (560h14m39.326304743s) [noise=true , address=45.91.116.10:4084 , piece_id=4RJ5ZW4W6CZO7PU5MMQWZSAM3PT2J3CXJ56LQVHF4S7XREVSJWIQ, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455310047 (560h14m39.326393092s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53538->145.79.227.23:4547: use of closed network connection Dec 28 23:43:39.455319585 (560h14m39.32640263s) [noise=true , address=145.79.227.23:4547 , piece_id=MEWIDQUI5FFFGCE6JS36T3XVIOHCE6NXUWI7G56APBYV7TRF73HQ, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455587786 (560h14m39.326670821s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 23:43:39.455614255 (560h14m39.32669729s) [noise=true , address=145.79.227.44:4195 , piece_id=KFAQAZC6NLSTVH6NXH7HXG6XYK4HG4HQOG4E53EVBHNX54QTQZ6Q, seg_pos={0 0}, upload=491167]: Long tail cancellation due to optimal threshold Dec 28 23:43:39.455638041 (560h14m39.326721076s) [seg_pos={0 0}, upload=491167]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:43:39.455759512 (560h14m39.326842557s) [upload=491167]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:43:39.455765264 (560h14m39.326848299s) [upload=491167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:43:39.750546135 (560h14m39.62162917s) [upload=491167]: Done uploading segments... Dec 28 23:43:39.750565742 (560h14m39.621648777s) [upload=491167]: Upload finished. err: <nil> Duration: 30.003017086s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.498292597s trace: https://jaeger.dev.storj.io/trace/52ecb9f526c189a5 FAIL (2025-12-28 23:43:29 UTC) orbiter warning: context deadline exceeded Duration: 3.49832117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 27.881726768s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:43:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 27.881752447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.4767803s trace: https://jaeger.dev.storj.io/trace/52ecb9f526c189a5 FAIL (2025-12-28 23:43:00 UTC) orbiter warning: context deadline exceeded Duration: 3.476809615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 22.188494323s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:42:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 22.188523639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................X..X.X....X.....X.....X......] duration: 6.101182596s trace: https://jaeger.dev.storj.io/trace/610a0767a6ac8a82 FAIL (2025-12-28 23:42:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :38:08.566668213 (560h9m8.437751258s) [upload=491088]: Uploading segments... Dec 28 23:38:08.566828227 (560h9m8.437911262s) [upload=491088]: Got next segment. Inline: false Dec 28 23:38:08.566901878 (560h9m8.437984913s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:38:08.566910575 (560h9m8.43799362s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:38:08.808608158 (560h9m8.679691193s) [seg_pos={0 0}, upload=491088]: Begin upload segment... Dec 28 23:38:08.808652772 (560h9m8.679735817s) [seg_pos={0 0}, upload=491088]: Done begin upload segment. Dec 28 23:38:08.808681657 (560h9m8.679764692s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.808694422 (560h9m8.679777457s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809309172 (560h9m8.680392207s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809455090 (560h9m8.680538135s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809908725 (560h9m8.68099176s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810644976 (560h9m8.681728012s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810737783 (560h9m8.681820848s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811330031 (560h9m8.682413076s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811420533 (560h9m8.682503578s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812219645 (560h9m8.68330268s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812942451 (560h9m8.684025486s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.813880007 (560h9m8.684963052s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.814466834 (560h9m8.685549879s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.815245558 (560h9m8.686328603s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816461854 (560h9m8.687544899s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816872154 (560h9m8.687955189s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:10.522010719 (560h9m10.393093764s) [upload=491088]: Next returned nil segment Dec 28 23:38:10.522095360 (560h9m10.393178425s) [upload=491088]: Waiting for error group managing segments... Dec 28 23:38:10.801884312 (560h9m10.672967357s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.804835431 (560h9m10.675918466s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.815328654 (560h9m10.686411689s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818677231 (560h9m10.689760276s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818864527 (560h9m10.689947562s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.819164869 (560h9m10.690247914s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.835374396 (560h9m10.706457442s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.843879313 (560h9m10.714962358s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.926650253 (560h9m10.797733288s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.027400883 (560h9m10.898483928s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.053900246 (560h9m10.924983291s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.149733491 (560h9m11.020816536s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291193012 (560h9m11.162276068s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291213892 (560h9m11.162296937s) [seg_pos={0 0}, upload=491088]: Segment reached optimal threshold of 13 pieces. Dec 28 23:38:11.291372985 (560h9m11.16245604s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 28 23:38:11.291378616 (560h9m11.162461651s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291583426 (560h9m11.162666461s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection Dec 28 23:38:11.291590158 (560h9m11.162673203s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291600849 (560h9m11.162683914s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection Dec 28 23:38:11.291621248 (560h9m11.162704313s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291689387 (560h9m11.162772442s) [seg_pos={0 0}, upload=491088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:38:11.291766304 (560h9m11.162849349s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:38:11.291770873 (560h9m11.162853908s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:38:11.586972335 (560h9m11.45805538s) [upload=491088]: Done uploading segments... Dec 28 23:38:11.586995729 (560h9m11.458078774s) [upload=491088]: Upload finished. err: <nil> Duration: 6.10121616s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.529217046s trace: https://jaeger.dev.storj.io/trace/52ecb9f526c189a5 FAIL (2025-12-28 23:42:25 UTC) orbiter warning: context deadline exceeded Duration: 3.529239359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 15.382351487s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:41:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 15.382380543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 24.419648152s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:41:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 24.419666186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................X..X.X....X.....X.....X.....] duration: 5.838239554s trace: https://jaeger.dev.storj.io/trace/610a0767a6ac8a82 FAIL (2025-12-28 23:41:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :38:08.566668213 (560h9m8.437751258s) [upload=491088]: Uploading segments... Dec 28 23:38:08.566828227 (560h9m8.437911262s) [upload=491088]: Got next segment. Inline: false Dec 28 23:38:08.566901878 (560h9m8.437984913s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:38:08.566910575 (560h9m8.43799362s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:38:08.808608158 (560h9m8.679691193s) [seg_pos={0 0}, upload=491088]: Begin upload segment... Dec 28 23:38:08.808652772 (560h9m8.679735817s) [seg_pos={0 0}, upload=491088]: Done begin upload segment. Dec 28 23:38:08.808681657 (560h9m8.679764692s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.808694422 (560h9m8.679777457s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809309172 (560h9m8.680392207s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809455090 (560h9m8.680538135s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809908725 (560h9m8.68099176s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810644976 (560h9m8.681728012s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810737783 (560h9m8.681820848s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811330031 (560h9m8.682413076s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811420533 (560h9m8.682503578s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812219645 (560h9m8.68330268s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812942451 (560h9m8.684025486s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.813880007 (560h9m8.684963052s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.814466834 (560h9m8.685549879s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.815245558 (560h9m8.686328603s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816461854 (560h9m8.687544899s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816872154 (560h9m8.687955189s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:10.522010719 (560h9m10.393093764s) [upload=491088]: Next returned nil segment Dec 28 23:38:10.522095360 (560h9m10.393178425s) [upload=491088]: Waiting for error group managing segments... Dec 28 23:38:10.801884312 (560h9m10.672967357s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.804835431 (560h9m10.675918466s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.815328654 (560h9m10.686411689s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818677231 (560h9m10.689760276s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818864527 (560h9m10.689947562s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.819164869 (560h9m10.690247914s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.835374396 (560h9m10.706457442s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.843879313 (560h9m10.714962358s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.926650253 (560h9m10.797733288s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.027400883 (560h9m10.898483928s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.053900246 (560h9m10.924983291s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.149733491 (560h9m11.020816536s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291193012 (560h9m11.162276068s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291213892 (560h9m11.162296937s) [seg_pos={0 0}, upload=491088]: Segment reached optimal threshold of 13 pieces. Dec 28 23:38:11.291372985 (560h9m11.16245604s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 28 23:38:11.291378616 (560h9m11.162461651s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291583426 (560h9m11.162666461s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection Dec 28 23:38:11.291590158 (560h9m11.162673203s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291600849 (560h9m11.162683914s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection Dec 28 23:38:11.291621248 (560h9m11.162704313s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291689387 (560h9m11.162772442s) [seg_pos={0 0}, upload=491088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:38:11.291766304 (560h9m11.162849349s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:38:11.291770873 (560h9m11.162853908s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:38:11.586972335 (560h9m11.45805538s) [upload=491088]: Done uploading segments... Dec 28 23:38:11.586995729 (560h9m11.458078774s) [upload=491088]: Upload finished. err: <nil> Duration: 5.838267307s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.569422829s trace: https://jaeger.dev.storj.io/trace/52ecb9f526c189a5 FAIL (2025-12-28 23:41:15 UTC) orbiter warning: context deadline exceeded Duration: 3.569450702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................X..X.X....X.....X.....X....] duration: 6.154825971s trace: https://jaeger.dev.storj.io/trace/610a0767a6ac8a82 FAIL (2025-12-28 23:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :38:08.566668213 (560h9m8.437751258s) [upload=491088]: Uploading segments... Dec 28 23:38:08.566828227 (560h9m8.437911262s) [upload=491088]: Got next segment. Inline: false Dec 28 23:38:08.566901878 (560h9m8.437984913s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:38:08.566910575 (560h9m8.43799362s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:38:08.808608158 (560h9m8.679691193s) [seg_pos={0 0}, upload=491088]: Begin upload segment... Dec 28 23:38:08.808652772 (560h9m8.679735817s) [seg_pos={0 0}, upload=491088]: Done begin upload segment. Dec 28 23:38:08.808681657 (560h9m8.679764692s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.808694422 (560h9m8.679777457s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809309172 (560h9m8.680392207s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809455090 (560h9m8.680538135s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809908725 (560h9m8.68099176s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810644976 (560h9m8.681728012s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810737783 (560h9m8.681820848s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811330031 (560h9m8.682413076s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811420533 (560h9m8.682503578s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812219645 (560h9m8.68330268s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812942451 (560h9m8.684025486s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.813880007 (560h9m8.684963052s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.814466834 (560h9m8.685549879s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.815245558 (560h9m8.686328603s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816461854 (560h9m8.687544899s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816872154 (560h9m8.687955189s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:10.522010719 (560h9m10.393093764s) [upload=491088]: Next returned nil segment Dec 28 23:38:10.522095360 (560h9m10.393178425s) [upload=491088]: Waiting for error group managing segments... Dec 28 23:38:10.801884312 (560h9m10.672967357s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.804835431 (560h9m10.675918466s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.815328654 (560h9m10.686411689s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818677231 (560h9m10.689760276s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818864527 (560h9m10.689947562s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.819164869 (560h9m10.690247914s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.835374396 (560h9m10.706457442s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.843879313 (560h9m10.714962358s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.926650253 (560h9m10.797733288s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.027400883 (560h9m10.898483928s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.053900246 (560h9m10.924983291s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.149733491 (560h9m11.020816536s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291193012 (560h9m11.162276068s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291213892 (560h9m11.162296937s) [seg_pos={0 0}, upload=491088]: Segment reached optimal threshold of 13 pieces. Dec 28 23:38:11.291372985 (560h9m11.16245604s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 28 23:38:11.291378616 (560h9m11.162461651s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291583426 (560h9m11.162666461s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection Dec 28 23:38:11.291590158 (560h9m11.162673203s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291600849 (560h9m11.162683914s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection Dec 28 23:38:11.291621248 (560h9m11.162704313s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291689387 (560h9m11.162772442s) [seg_pos={0 0}, upload=491088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:38:11.291766304 (560h9m11.162849349s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:38:11.291770873 (560h9m11.162853908s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:38:11.586972335 (560h9m11.45805538s) [upload=491088]: Done uploading segments... Dec 28 23:38:11.586995729 (560h9m11.458078774s) [upload=491088]: Upload finished. err: <nil> Duration: 6.154846348s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.535616962s trace: https://jaeger.dev.storj.io/trace/52ecb9f526c189a5 FAIL (2025-12-28 23:40:34 UTC) orbiter warning: context deadline exceeded Duration: 3.535781917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 15.284102856s trace: https://jaeger.dev.storj.io/trace/3b4b0b46993d9e04 FAIL (2025-12-28 23:40:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== LK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751582126 (559h58m33.622665221s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.751740378 (559h58m33.622823413s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752934551 (559h58m33.624017586s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.752962855 (559h58m33.6240459s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754182999 (559h58m33.625266044s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.754243433 (559h58m33.625326468s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755265490 (559h58m33.626348525s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.755422608 (559h58m33.626505643s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756433984 (559h58m33.627517029s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.756585483 (559h58m33.627668518s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.757641664 (559h58m33.628724699s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.759106804 (559h58m33.630189849s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:33.760795640 (559h58m33.631878675s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Uploading piece... Dec 28 23:27:35.229170705 (559h58m35.10025374s) [upload=490936]: Next returned nil segment Dec 28 23:27:35.229184070 (559h58m35.100267105s) [upload=490936]: Waiting for error group managing segments... Dec 28 23:27:35.689748882 (559h58m35.560831917s) [noise=true , address=145.79.227.4:4213 , piece_id=HY6JVZNH7KDC77NZVCB3YC6ZTM44AV3HAOSGTE4DPRFC53KGD4UQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.801525570 (559h58m35.672608605s) [noise=true , address=77.90.31.10:4537 , piece_id=WNJVJYG7GQHIHQTGIF2ASMV44P3RJFVNHE3PMWBLVSIHY6U6BOPQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.831711448 (559h58m35.702794483s) [noise=true , address=77.90.31.12:4607 , piece_id=MOH2PCOJAWVORSJXZN5ATUG2BY7AGUNJJPQXKLP7L4GZ3QMKXX4A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.840776860 (559h58m35.711859895s) [noise=true , address=77.90.31.13:4646 , piece_id=GQGGNPCMZM3GCYMIGRAAK4MOXWDHGB5XMLLNIORQIBJHZ2UJOJKQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.912090892 (559h58m35.783173927s) [noise=true , address=5.231.73.3:4026 , piece_id=6SHNHK53BDQVGXISUGOO5ULPDRBFJHI4ZPZN6GZ3PJXLKXRCOUTQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:35.932334218 (559h58m35.803417253s) [noise=true , address=145.79.227.47:4277 , piece_id=XWDGKHR45I7MLUICXK7H4733QXF3NSC6NWFAA6ER45V5VD3DDM2Q, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.479614398 (559h58m36.350697593s) [noise=true , address=145.79.227.30:4787 , piece_id=KJIYK5GYQJOMAYZNAHY3C75KM6W25OLPXW55HUJM26J4TUKVWPRQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.496525048 (559h58m36.367608083s) [noise=true , address=77.90.31.14:4684 , piece_id=ROMDRQ3ALCAOMX6UDVHO2TSNA26XRAG42SA2KWKWKPXGTCYOH4GA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.509559748 (559h58m36.380642783s) [noise=true , address=77.90.31.18:4821 , piece_id=7EEKNLKVCMAR2X452DVPF5H4HV7PHV6KZ4LGYYSGQXOY6LB24YMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.558796932 (559h58m36.429879967s) [noise=true , address=145.79.227.34:4940 , piece_id=L74UCRSOUBUEEOAPRIEGLMPPBRFY5VUAB6LFQHWNBN7NB4DJG6UA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.562046140 (559h58m36.433129185s) [noise=true , address=145.79.227.44:4222 , piece_id=BUYN55YORL7MEX5YQGTDJGNKLK77IISFFY4HPIBMYYXZOQQXRWAA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.574372210 (559h58m36.445455245s) [noise=true , address=77.90.31.17:4775 , piece_id=T377U23CIFKXXHVO2QBK2344MEEMM4ND57J7IQE3ZMPN7ICR6DGA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.646469704 (559h58m36.517552749s) [noise=true , address=145.79.227.43:4180 , piece_id=I6Q223R4J4OXSF52KS7ROLKEDFZWQ3JLTJYLULE3E4YDNOMNEWIA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669784909 (559h58m36.540867944s) [noise=true , address=145.79.227.33:4892 , piece_id=RVHT3GZ7GGSIMMSQTNZHIUELDF2K7WVXHK6WGXU7M2ODDIAX262A, seg_pos={0 0}, upload=490936]: Done uploading piece. err: <nil> Dec 28 23:27:36.669897153 (559h58m36.540980188s) [seg_pos={0 0}, upload=490936]: Segment reached optimal threshold of 14 pieces. Dec 28 23:27:36.670107704 (559h58m36.541190749s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->145.79.227.16:4935: use of closed network connection Dec 28 23:27:36.670114738 (559h58m36.541197773s) [noise=true , address=145.79.227.16:4935 , piece_id=2KHL5EMPL5K6NZYYITU23GJEK6W7AXF6L3CENPALSAIRDURR22IQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670185232 (559h58m36.541268287s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 28 23:27:36.670220399 (559h58m36.541303444s) [noise=true , address=145.79.227.40:4075 , piece_id=RNDQKJMSRD2COFXBBE57D3YTIP4W5IMXHP3AFXK7ZDSBJVYZWUYA, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670322273 (559h58m36.541405308s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->64.49.13.169:4716: use of closed network connection Dec 28 23:27:36.670328294 (559h58m36.541411339s) [noise=true , address=64.49.13.169:4716 , piece_id=HWN5LMAFEN3JPGS2U3I56IYS4Q4MKFPTLGDXOZYDE5II2MVF7GMQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670428044 (559h58m36.54151108s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cDeVufqmqEpR5hjFrhgFmsS7TYeTnhEx3Xyet9ck39sj7Eeiz): protocol: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38942->145.79.227.23:4563: use of closed network connection Dec 28 23:27:36.670433044 (559h58m36.541516079s) [noise=true , address=145.79.227.23:4563 , piece_id=L4TLEC5GAPLBLTVDOK2CCJYLHJ7X5D6QABIW6PTIMKOYERENE7JQ, seg_pos={0 0}, upload=490936]: Long tail cancellation due to optimal threshold Dec 28 23:27:36.670545428 (559h58m36.541628473s) [seg_pos={0 0}, upload=490936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 23:27:36.670643625 (559h58m36.54172667s) [upload=490936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:27:36.670650377 (559h58m36.541733413s) [upload=490936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:27:36.958444739 (559h58m36.829527774s) [upload=490936]: Done uploading segments... Dec 28 23:27:36.958461782 (559h58m36.829544817s) [upload=490936]: Upload finished. err: <nil> Duration: 15.284133085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 5.000810204s trace: https://jaeger.dev.storj.io/trace/52ecb9f526c189a5 FAIL (2025-12-28 23:40:03 UTC) orbiter warning: context deadline exceeded Duration: 5.000839852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................X..X.X....X.....X.....X...] duration: 7.038121168s trace: https://jaeger.dev.storj.io/trace/610a0767a6ac8a82 FAIL (2025-12-28 23:40:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :38:08.566668213 (560h9m8.437751258s) [upload=491088]: Uploading segments... Dec 28 23:38:08.566828227 (560h9m8.437911262s) [upload=491088]: Got next segment. Inline: false Dec 28 23:38:08.566901878 (560h9m8.437984913s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:38:08.566910575 (560h9m8.43799362s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:38:08.808608158 (560h9m8.679691193s) [seg_pos={0 0}, upload=491088]: Begin upload segment... Dec 28 23:38:08.808652772 (560h9m8.679735817s) [seg_pos={0 0}, upload=491088]: Done begin upload segment. Dec 28 23:38:08.808681657 (560h9m8.679764692s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.808694422 (560h9m8.679777457s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809309172 (560h9m8.680392207s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809455090 (560h9m8.680538135s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809908725 (560h9m8.68099176s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810644976 (560h9m8.681728012s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810737783 (560h9m8.681820848s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811330031 (560h9m8.682413076s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811420533 (560h9m8.682503578s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812219645 (560h9m8.68330268s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812942451 (560h9m8.684025486s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.813880007 (560h9m8.684963052s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.814466834 (560h9m8.685549879s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.815245558 (560h9m8.686328603s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816461854 (560h9m8.687544899s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816872154 (560h9m8.687955189s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:10.522010719 (560h9m10.393093764s) [upload=491088]: Next returned nil segment Dec 28 23:38:10.522095360 (560h9m10.393178425s) [upload=491088]: Waiting for error group managing segments... Dec 28 23:38:10.801884312 (560h9m10.672967357s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.804835431 (560h9m10.675918466s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.815328654 (560h9m10.686411689s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818677231 (560h9m10.689760276s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818864527 (560h9m10.689947562s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.819164869 (560h9m10.690247914s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.835374396 (560h9m10.706457442s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.843879313 (560h9m10.714962358s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.926650253 (560h9m10.797733288s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.027400883 (560h9m10.898483928s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.053900246 (560h9m10.924983291s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.149733491 (560h9m11.020816536s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291193012 (560h9m11.162276068s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291213892 (560h9m11.162296937s) [seg_pos={0 0}, upload=491088]: Segment reached optimal threshold of 13 pieces. Dec 28 23:38:11.291372985 (560h9m11.16245604s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 28 23:38:11.291378616 (560h9m11.162461651s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291583426 (560h9m11.162666461s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection Dec 28 23:38:11.291590158 (560h9m11.162673203s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291600849 (560h9m11.162683914s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection Dec 28 23:38:11.291621248 (560h9m11.162704313s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291689387 (560h9m11.162772442s) [seg_pos={0 0}, upload=491088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:38:11.291766304 (560h9m11.162849349s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:38:11.291770873 (560h9m11.162853908s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:38:11.586972335 (560h9m11.45805538s) [upload=491088]: Done uploading segments... Dec 28 23:38:11.586995729 (560h9m11.458078774s) [upload=491088]: Upload finished. err: <nil> Duration: 7.0381544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................X..X.X....X.....X.....X..] duration: 5.94606663s trace: https://jaeger.dev.storj.io/trace/610a0767a6ac8a82 FAIL (2025-12-28 23:39:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :38:08.566668213 (560h9m8.437751258s) [upload=491088]: Uploading segments... Dec 28 23:38:08.566828227 (560h9m8.437911262s) [upload=491088]: Got next segment. Inline: false Dec 28 23:38:08.566901878 (560h9m8.437984913s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 23:38:08.566910575 (560h9m8.43799362s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 23:38:08.808608158 (560h9m8.679691193s) [seg_pos={0 0}, upload=491088]: Begin upload segment... Dec 28 23:38:08.808652772 (560h9m8.679735817s) [seg_pos={0 0}, upload=491088]: Done begin upload segment. Dec 28 23:38:08.808681657 (560h9m8.679764692s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.808694422 (560h9m8.679777457s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809309172 (560h9m8.680392207s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809455090 (560h9m8.680538135s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.809908725 (560h9m8.68099176s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810644976 (560h9m8.681728012s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.810737783 (560h9m8.681820848s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811330031 (560h9m8.682413076s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.811420533 (560h9m8.682503578s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812219645 (560h9m8.68330268s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.812942451 (560h9m8.684025486s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.813880007 (560h9m8.684963052s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.814466834 (560h9m8.685549879s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.815245558 (560h9m8.686328603s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816461854 (560h9m8.687544899s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:08.816872154 (560h9m8.687955189s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Uploading piece... Dec 28 23:38:10.522010719 (560h9m10.393093764s) [upload=491088]: Next returned nil segment Dec 28 23:38:10.522095360 (560h9m10.393178425s) [upload=491088]: Waiting for error group managing segments... Dec 28 23:38:10.801884312 (560h9m10.672967357s) [noise=true , address=145.79.227.52:4446 , piece_id=FHT6NCCQ3E6CJM4GJDKHAHQRLPVC4N2FVUVQFJ52NPEQJAOU76ZA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.804835431 (560h9m10.675918466s) [noise=true , address=145.79.227.48:4301 , piece_id=LNONF5L3TCKU7RMW5P5FPEALB3R3LKFNMFD2UVPTH3SIKCFBKIUA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.815328654 (560h9m10.686411689s) [noise=true , address=145.79.227.11:4729 , piece_id=LWDEXEM2YSKKN5YKQQSDCUOUUOROOEQ6GUPKOW5C27Z5WMJ2EZTQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818677231 (560h9m10.689760276s) [noise=true , address=145.79.227.3:4150 , piece_id=ESH3PQ35LEN7OXR6HYWDZ4MAJKNQE3XNU7ZIPYSSRGDSXMSFP46A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.818864527 (560h9m10.689947562s) [noise=true , address=145.79.227.50:4374 , piece_id=RAGT5CW4SYN5VCZQWUB4CTIJKG7LN62NJ6TEFALUDTBRRCLTSGVA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.819164869 (560h9m10.690247914s) [noise=true , address=145.79.227.42:4121 , piece_id=ROPAMM24QP2M7SZ7MGNB4LWG5NGONY6ZO7Z6GFTJFEPMEI5OXFRQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.835374396 (560h9m10.706457442s) [noise=true , address=145.79.227.23:4547 , piece_id=W3K6GOVHYQRPEPCJCZ3RQMKHWIKVSADA36DFAY4VTEROPZJ4SAGA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.843879313 (560h9m10.714962358s) [noise=true , address=145.79.227.36:4518 , piece_id=YL3622GY4TIIGV5T6VLETTHWW6PHPLVRTHWA6NWPGPRB64K5CDGQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:10.926650253 (560h9m10.797733288s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2D5BC5B3RNQOJLNSI7SE2A7DRTOJ2X5Y4XTQ3CY6P2WASNT34SQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.027400883 (560h9m10.898483928s) [noise=true , address=145.79.227.5:4221 , piece_id=Q7GMALB7DT6XAVUPH27KCFTIEP7ZRC6GGHMVQHACHVKXT6XVF3KQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.053900246 (560h9m10.924983291s) [noise=true , address=45.91.116.6:4975 , piece_id=DVCCHQOSZ6OKTX4RDV4A74X5J7BJJLAVGVIAHATR4NPTH25XIGIQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.149733491 (560h9m11.020816536s) [noise=true , address=45.91.116.11:4119 , piece_id=6PLQXRGUYTQLLCIC3QJDHCPTVISQAUAFUIQCKSWRTYXDISXQDH2Q, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291193012 (560h9m11.162276068s) [noise=true , address=77.90.31.12:4590 , piece_id=HCJWYFWATKWOMJ44E3EUA4FK2X6YJKKPMKJR24XDEZY3SHHWFLDQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: <nil> Dec 28 23:38:11.291213892 (560h9m11.162296937s) [seg_pos={0 0}, upload=491088]: Segment reached optimal threshold of 13 pieces. Dec 28 23:38:11.291372985 (560h9m11.16245604s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 28 23:38:11.291378616 (560h9m11.162461651s) [noise=true , address=145.79.227.4:4185 , piece_id=UTDUERCTSXYIVUOIE4KTV326OF4AKOHZQMARNINS7XNPOE6C2EBQ, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291583426 (560h9m11.162666461s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35088->45.91.116.4:4940: use of closed network connection Dec 28 23:38:11.291590158 (560h9m11.162673203s) [noise=true , address=45.91.116.4:4940 , piece_id=4VIY6PS5Z52D3QC4S3LGIM4MW4DYVOAT3UOPKWLSLBA2CZBHUC5A, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291600849 (560h9m11.162683914s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46250->145.79.227.18:4006: use of closed network connection Dec 28 23:38:11.291621248 (560h9m11.162704313s) [noise=true , address=145.79.227.18:4006 , piece_id=R3DDQOFG6B52TGYFD2KNDKJOQQKTBV42JQDP4E5LDHVT77336KOA, seg_pos={0 0}, upload=491088]: Long tail cancellation due to optimal threshold Dec 28 23:38:11.291689387 (560h9m11.162772442s) [seg_pos={0 0}, upload=491088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 23:38:11.291766304 (560h9m11.162849349s) [upload=491088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 23:38:11.291770873 (560h9m11.162853908s) [upload=491088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 23:38:11.586972335 (560h9m11.45805538s) [upload=491088]: Done uploading segments... Dec 28 23:38:11.586995729 (560h9m11.458078774s) [upload=491088]: Upload finished. err: <nil> Duration: 5.946094021s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)