Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..XX....X...XXXX...X..X..........X..X.X.X.X..X............] duration: 5.906624867s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:56:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 5.906649806s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.472433156s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:56:14 UTC) orbiter warning: context deadline exceeded Duration: 3.472462023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X...........] duration: 6.01904287s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:55:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 6.019070764s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.547108283s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:55:19 UTC) orbiter warning: context deadline exceeded Duration: 3.547135304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X..........] duration: 14.015353862s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:55:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 14.015383499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X.........] duration: 5.970055477s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 5.970073333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X........] duration: 23.154329678s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:54:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 23.154356701s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.541926074s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:54:16 UTC) orbiter warning: context deadline exceeded Duration: 3.541955531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X.......] duration: 5.907888401s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:53:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 5.907918941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X......] duration: 25.575093237s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:53:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 25.575121672s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.585468716s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:53:07 UTC) orbiter warning: context deadline exceeded Duration: 3.585500826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X.....] duration: 7.139149053s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:52:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 7.139264123s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.514152664s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:51:54 UTC) orbiter warning: context deadline exceeded Duration: 3.514184495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X....] duration: 5.9282427s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:51:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 5.928259594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X...] duration: 29.156718615s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:51:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 29.156747821s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.589936234s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:50:43 UTC) orbiter warning: context deadline exceeded Duration: 3.589962083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.529626255s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:50:21 UTC) orbiter warning: context deadline exceeded Duration: 3.52966642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X..] duration: 21.510090329s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:50:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 21.51013839s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.508219952s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:49:44 UTC) orbiter warning: context deadline exceeded Duration: 3.508243496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X.] duration: 5.634477488s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:48:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 5.634505333s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.594433097s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:48:55 UTC) orbiter warning: context deadline exceeded Duration: 3.594458886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.624858774s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:48:30 UTC) orbiter warning: context deadline exceeded Duration: 3.625017747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [......X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..X] duration: 30.003316841s trace: https://jaeger.dev.storj.io/trace/4f51a513fb331141 FAIL (2025-12-27 19:48:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 27 19:47:48.242902465 (532h18m48.1139855s) [upload=466840]: Got next segment. Inline: false Dec 27 19:47:48.242930649 (532h18m48.114013684s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:47:48.242935208 (532h18m48.114018243s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:47:48.492981741 (532h18m48.364064776s) [seg_pos={0 0}, upload=466840]: Begin upload segment... Dec 27 19:47:48.493024673 (532h18m48.364107718s) [seg_pos={0 0}, upload=466840]: Done begin upload segment. Dec 27 19:47:48.493048549 (532h18m48.364131584s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493056053 (532h18m48.364139088s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493449091 (532h18m48.364532126s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.493925238 (532h18m48.365008273s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494279492 (532h18m48.365362547s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494617987 (532h18m48.365701022s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.494939499 (532h18m48.366022534s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.495285508 (532h18m48.366368553s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.496514808 (532h18m48.367597853s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.497597812 (532h18m48.368680847s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.501348473 (532h18m48.372431508s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.504865410 (532h18m48.375948445s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.505905160 (532h18m48.376988195s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.506848085 (532h18m48.37793112s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.507081910 (532h18m48.378164956s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:48.509611427 (532h18m48.380694462s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Uploading piece... Dec 27 19:47:50.209427783 (532h18m50.080510818s) [upload=466840]: Next returned nil segment Dec 27 19:47:50.209447942 (532h18m50.080530977s) [upload=466840]: Waiting for error group managing segments... Dec 27 19:47:50.504063253 (532h18m50.375146308s) [noise=true , address=145.79.227.50:4374 , piece_id=6VBNDMAX7VCS2K375FMPGERJ3VTKX5J7ACEN3RBV2BADLWL4Z2NA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.505863782 (532h18m50.376946837s) [noise=true , address=145.79.227.10:4402 , piece_id=Q23URXKTSKCMW3F7YNWQNSVMYGTNTTYNZFNUFNFTPQNSGS2YQXWQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.508153260 (532h18m50.379236305s) [noise=true , address=145.79.227.2:4113 , piece_id=SZJRK3BIPPHMBRMPIA5OT433PACY6M3U4GRVSPLER2IHLP2WI4XQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.523456220 (532h18m50.394539265s) [noise=true , address=145.79.227.46:4601 , piece_id=YL7QY5U2BRXVD4XPA5PRQP7J7KZN2XAZIH6H7G2BGCOZNUZ3B72A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.527331690 (532h18m50.398414735s) [noise=true , address=145.79.227.25:4618 , piece_id=5BFPRBNL57GDYX65KENXCUE7CT4HPTNYRH6CPKMZ3NUHO35KGJJA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.555653369 (532h18m50.426736404s) [noise=true , address=145.79.227.39:4015 , piece_id=SU35VKPTII4LEVE3P2QNVXEM2G5HCZ7R253ENB4XZOQPCF6RA52Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.743742063 (532h18m50.614825099s) [noise=true , address=45.91.116.11:4118 , piece_id=NESKOAXOUG6VYBO2HXPWY7RJ4BVRXMSKKBAIKBAITRJT7BJSKY2Q, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.761499618 (532h18m50.632582653s) [noise=true , address=45.91.116.8:4045 , piece_id=RYWGJSR5C72DCKRB4KR4RN4QLF5GIMZUED5U7GRT7RWIVLRIVUQQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.786166037 (532h18m50.657249082s) [noise=true , address=77.90.31.13:4627 , piece_id=JDBIYHYUYYJVMYCP7PV4CGXPHNPFJT2JFSHKX2R6ELCEF43SRKEQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:50.788902437 (532h18m50.659985472s) [noise=true , address=45.91.116.4:4939 , piece_id=IU7PZ3SBAIADWTFD5PHOUELKNVYE6JPOWJ63J2R2KCCBQJETRFLA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.453537421 (532h18m51.324620466s) [noise=true , address=45.91.116.10:4084 , piece_id=BMVNU66RUVTOK7ZF53BLLU6KHXYGKYAWBZNVFG6LLSNNGCE5LZWA, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.465463862 (532h18m51.336546897s) [noise=true , address=45.91.116.14:4192 , piece_id=WAWTFV5GMPD2AK2PM3ANCEMXWTYSUUPDKEZA3XIJJ6SPY4PFA67A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707107195 (532h18m51.57819026s) [noise=true , address=45.91.116.9:4263 , piece_id=3O2WZHRZIGWVCL3KW5KO4ZQECMNRJKQUHJN4UF23D3BDXFOOFN2A, seg_pos={0 0}, upload=466840]: Done uploading piece. err: <nil> Dec 27 19:47:51.707134577 (532h18m51.578217612s) [seg_pos={0 0}, upload=466840]: Segment reached optimal threshold of 13 pieces. Dec 27 19:47:51.707403309 (532h18m51.578486344s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34592->145.79.227.6:4257: use of closed network connection Dec 27 19:47:51.707414510 (532h18m51.578497545s) [noise=true , address=145.79.227.6:4257 , piece_id=DAHES434JB5CUAZGDYURAXEWLQOFQVKFW4ZFZN4UHSXQG6UQAEMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707646141 (532h18m51.578729176s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 19:47:51.707652814 (532h18m51.578735849s) [noise=true , address=145.79.227.44:4195 , piece_id=IGF6U4NG5LFQJCJRELY6TNLWVJPX34ZE3IKSCNJUQ7OKYXJ2LKRQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707738947 (532h18m51.578821982s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.51:4410: use of closed network connection Dec 27 19:47:51.707759968 (532h18m51.578843013s) [noise=true , address=145.79.227.51:4410 , piece_id=N5A2BIIL4AVLYZDUWLE72YIOYG3ZY4Q6W4MO2U7HWT6RX3AH3QMQ, seg_pos={0 0}, upload=466840]: Long tail cancellation due to optimal threshold Dec 27 19:47:51.707820894 (532h18m51.578903929s) [seg_pos={0 0}, upload=466840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:47:51.707889835 (532h18m51.57897288s) [upload=466840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:47:51.707894023 (532h18m51.578977058s) [upload=466840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:47:52.003140335 (532h18m51.87422339s) [upload=466840]: Done uploading segments... Dec 27 19:47:52.003163489 (532h18m51.874246534s) [upload=466840]: Upload finished. err: <nil> Duration: 30.003339665s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.406506311s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:47:44 UTC) orbiter warning: context deadline exceeded Duration: 3.406533973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.399159752s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:47:00 UTC) orbiter warning: context deadline exceeded Duration: 3.399187645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X..] duration: 5.808519666s trace: https://jaeger.dev.storj.io/trace/6cb8bf31d0386b6 FAIL (2025-12-27 19:46:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:45:30.180765025 (532h16m30.05184807s) [upload=466807]: Starting upload Dec 27 19:45:30.180854125 (532h16m30.05193716s) [upload=466807]: Done starting upload Dec 27 19:45:30.180899090 (532h16m30.051982125s) [upload=466807]: Uploading segments... Dec 27 19:45:30.181034428 (532h16m30.052117473s) [upload=466807]: Got next segment. Inline: false Dec 27 19:45:30.181065147 (532h16m30.052148192s) [upload=466807]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:45:30.181069295 (532h16m30.05215234s) [upload=466807]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:45:30.425047633 (532h16m30.296130668s) [seg_pos={0 0}, upload=466807]: Begin upload segment... Dec 27 19:45:30.425112035 (532h16m30.296195081s) [seg_pos={0 0}, upload=466807]: Done begin upload segment. Dec 27 19:45:30.425175927 (532h16m30.296258962s) [noise=true , address=145.79.227.4:4187 , piece_id=IDQSOPGS436FNQ2MH5HFDUALWXEDVEPAV7TB2RXC7IUQHTQFWQYQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425214080 (532h16m30.296297125s) [noise=true , address=77.90.31.13:4629 , piece_id=DLVRLXWLPV5YK4MU3NUYVXESMPK5DJHL46XMNZ3Q6EPPXW5N6TOA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425755762 (532h16m30.296838797s) [noise=true , address=145.79.227.15:4876 , piece_id=E5Q2BC4MK3NJO5O3XJHO7SQKFLZXNNWPH2MEAS7U4NU4P3EC3SXA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425844220 (532h16m30.296927255s) [noise=true , address=145.79.227.51:4411 , piece_id=7LTOU2XSG5OBBLCFMIWTGU6EHVDU43WQD43OVZUU5LBSCXZC7MYQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.426224223 (532h16m30.297307268s) [noise=true , address=145.79.227.31:4799 , piece_id=3HO72R76OZWWSFUYF622GB6OOZTFQSFJPBFPZDM4YYJAQP2PRQQA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.426722041 (532h16m30.297805076s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427080665 (532h16m30.29816371s) [noise=true , address=145.79.227.38:4978 , piece_id=DKHGAP3ALG7DTM2CUHKPP3RGHHYO6IYW3Q46GUW4YUZBIC3GYMFQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427392148 (532h16m30.298475193s) [noise=true , address=145.79.227.5:4223 , piece_id=LXAK2RJRU6TE7IDTRFIV5CZSZHK5EVYZ6LRG4ZGJO45MRXLRBLYA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427730662 (532h16m30.298813707s) [noise=true , address=45.91.116.12:4227 , piece_id=LQTDZVZDXLCJCC7BT4MDIQY6O7C7EWXAXB57FYQHF7INIA2Q7MRA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428123290 (532h16m30.299206335s) [noise=true , address=145.79.227.10:4403 , piece_id=G7MJF4LCYZIOJZLEO7ERE3INJWJZ5JZ3S3GRIC52TNVJDXLUY4HA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428610367 (532h16m30.299693412s) [noise=true , address=145.79.227.22:4510 , piece_id=T5TBMCQIFOY442EXOZT3XXIB46HJT2ISE5ILBGDI3JQDZ5R6GXIQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428914897 (532h16m30.299997932s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429424056 (532h16m30.300507092s) [noise=true , address=145.79.227.52:4446 , piece_id=ZI4I5HHEWVCDGHK5PIWMA55LTVRJQOYZR6RQWN7T3MZ4WIK4UX6A, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429482578 (532h16m30.300565613s) [noise=true , address=45.91.116.9:4262 , piece_id=J4RMOAJ3SUWATSDLVKEUJPTAWLOZATWKNU56YQ2TO2F73IZ6CPIA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429918137 (532h16m30.301001172s) [noise=true , address=145.79.227.28:4727 , piece_id=TYET6VIFJTNPG2S2MQKYQKJO6SHRQKIQDYNBIZBNVY2F467L4ZCA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.430614593 (532h16m30.301697628s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:32.134839415 (532h16m32.00592245s) [upload=466807]: Next returned nil segment Dec 27 19:45:32.134851387 (532h16m32.005934432s) [upload=466807]: Waiting for error group managing segments... Dec 27 19:45:32.413663857 (532h16m32.284746892s) [noise=true , address=145.79.227.31:4799 , piece_id=3HO72R76OZWWSFUYF622GB6OOZTFQSFJPBFPZDM4YYJAQP2PRQQA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.420589615 (532h16m32.29167265s) [noise=true , address=145.79.227.15:4876 , piece_id=E5Q2BC4MK3NJO5O3XJHO7SQKFLZXNNWPH2MEAS7U4NU4P3EC3SXA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.421963530 (532h16m32.293046565s) [noise=true , address=145.79.227.4:4187 , piece_id=IDQSOPGS436FNQ2MH5HFDUALWXEDVEPAV7TB2RXC7IUQHTQFWQYQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.427098208 (532h16m32.298181253s) [noise=true , address=145.79.227.38:4978 , piece_id=DKHGAP3ALG7DTM2CUHKPP3RGHHYO6IYW3Q46GUW4YUZBIC3GYMFQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.435325524 (532h16m32.306408569s) [noise=true , address=145.79.227.22:4510 , piece_id=T5TBMCQIFOY442EXOZT3XXIB46HJT2ISE5ILBGDI3JQDZ5R6GXIQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.438898999 (532h16m32.309982034s) [noise=true , address=145.79.227.28:4727 , piece_id=TYET6VIFJTNPG2S2MQKYQKJO6SHRQKIQDYNBIZBNVY2F467L4ZCA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.442651824 (532h16m32.313734859s) [noise=true , address=145.79.227.51:4411 , piece_id=7LTOU2XSG5OBBLCFMIWTGU6EHVDU43WQD43OVZUU5LBSCXZC7MYQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.445326046 (532h16m32.316409112s) [noise=true , address=145.79.227.5:4223 , piece_id=LXAK2RJRU6TE7IDTRFIV5CZSZHK5EVYZ6LRG4ZGJO45MRXLRBLYA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.445426828 (532h16m32.316509864s) [noise=true , address=145.79.227.10:4403 , piece_id=G7MJF4LCYZIOJZLEO7ERE3INJWJZ5JZ3S3GRIC52TNVJDXLUY4HA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.467291953 (532h16m32.338374988s) [noise=true , address=145.79.227.52:4446 , piece_id=ZI4I5HHEWVCDGHK5PIWMA55LTVRJQOYZR6RQWN7T3MZ4WIK4UX6A, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.606582909 (532h16m32.477665954s) [noise=true , address=77.90.31.13:4629 , piece_id=DLVRLXWLPV5YK4MU3NUYVXESMPK5DJHL46XMNZ3Q6EPPXW5N6TOA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.645386708 (532h16m32.516469743s) [noise=true , address=45.91.116.9:4262 , piece_id=J4RMOAJ3SUWATSDLVKEUJPTAWLOZATWKNU56YQ2TO2F73IZ6CPIA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.689000717 (532h16m32.560083752s) [noise=true , address=45.91.116.12:4227 , piece_id=LQTDZVZDXLCJCC7BT4MDIQY6O7C7EWXAXB57FYQHF7INIA2Q7MRA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.689019143 (532h16m32.560102178s) [seg_pos={0 0}, upload=466807]: Segment reached optimal threshold of 13 pieces. Dec 27 19:45:32.689225004 (532h16m32.560308039s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689232549 (532h16m32.560315594s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689486783 (532h16m32.560569828s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689494537 (532h16m32.560577572s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689663990 (532h16m32.560747035s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689672015 (532h16m32.56075505s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689689349 (532h16m32.560772394s) [seg_pos={0 0}, upload=466807]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:45:32.689885522 (532h16m32.560968557s) [upload=466807]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:45:32.689893658 (532h16m32.560976693s) [upload=466807]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:45:32.982897359 (532h16m32.853980394s) [upload=466807]: Done uploading segments... Dec 27 19:45:32.982917677 (532h16m32.854000723s) [upload=466807]: Upload finished. err: <nil> Duration: 5.808549613s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.534808593s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:46:33 UTC) orbiter warning: context deadline exceeded Duration: 3.53483377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X.] duration: 5.844024535s trace: https://jaeger.dev.storj.io/trace/6cb8bf31d0386b6 FAIL (2025-12-27 19:46:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:45:30.180765025 (532h16m30.05184807s) [upload=466807]: Starting upload Dec 27 19:45:30.180854125 (532h16m30.05193716s) [upload=466807]: Done starting upload Dec 27 19:45:30.180899090 (532h16m30.051982125s) [upload=466807]: Uploading segments... Dec 27 19:45:30.181034428 (532h16m30.052117473s) [upload=466807]: Got next segment. Inline: false Dec 27 19:45:30.181065147 (532h16m30.052148192s) [upload=466807]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:45:30.181069295 (532h16m30.05215234s) [upload=466807]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:45:30.425047633 (532h16m30.296130668s) [seg_pos={0 0}, upload=466807]: Begin upload segment... Dec 27 19:45:30.425112035 (532h16m30.296195081s) [seg_pos={0 0}, upload=466807]: Done begin upload segment. Dec 27 19:45:30.425175927 (532h16m30.296258962s) [noise=true , address=145.79.227.4:4187 , piece_id=IDQSOPGS436FNQ2MH5HFDUALWXEDVEPAV7TB2RXC7IUQHTQFWQYQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425214080 (532h16m30.296297125s) [noise=true , address=77.90.31.13:4629 , piece_id=DLVRLXWLPV5YK4MU3NUYVXESMPK5DJHL46XMNZ3Q6EPPXW5N6TOA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425755762 (532h16m30.296838797s) [noise=true , address=145.79.227.15:4876 , piece_id=E5Q2BC4MK3NJO5O3XJHO7SQKFLZXNNWPH2MEAS7U4NU4P3EC3SXA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425844220 (532h16m30.296927255s) [noise=true , address=145.79.227.51:4411 , piece_id=7LTOU2XSG5OBBLCFMIWTGU6EHVDU43WQD43OVZUU5LBSCXZC7MYQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.426224223 (532h16m30.297307268s) [noise=true , address=145.79.227.31:4799 , piece_id=3HO72R76OZWWSFUYF622GB6OOZTFQSFJPBFPZDM4YYJAQP2PRQQA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.426722041 (532h16m30.297805076s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427080665 (532h16m30.29816371s) [noise=true , address=145.79.227.38:4978 , piece_id=DKHGAP3ALG7DTM2CUHKPP3RGHHYO6IYW3Q46GUW4YUZBIC3GYMFQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427392148 (532h16m30.298475193s) [noise=true , address=145.79.227.5:4223 , piece_id=LXAK2RJRU6TE7IDTRFIV5CZSZHK5EVYZ6LRG4ZGJO45MRXLRBLYA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427730662 (532h16m30.298813707s) [noise=true , address=45.91.116.12:4227 , piece_id=LQTDZVZDXLCJCC7BT4MDIQY6O7C7EWXAXB57FYQHF7INIA2Q7MRA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428123290 (532h16m30.299206335s) [noise=true , address=145.79.227.10:4403 , piece_id=G7MJF4LCYZIOJZLEO7ERE3INJWJZ5JZ3S3GRIC52TNVJDXLUY4HA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428610367 (532h16m30.299693412s) [noise=true , address=145.79.227.22:4510 , piece_id=T5TBMCQIFOY442EXOZT3XXIB46HJT2ISE5ILBGDI3JQDZ5R6GXIQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428914897 (532h16m30.299997932s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429424056 (532h16m30.300507092s) [noise=true , address=145.79.227.52:4446 , piece_id=ZI4I5HHEWVCDGHK5PIWMA55LTVRJQOYZR6RQWN7T3MZ4WIK4UX6A, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429482578 (532h16m30.300565613s) [noise=true , address=45.91.116.9:4262 , piece_id=J4RMOAJ3SUWATSDLVKEUJPTAWLOZATWKNU56YQ2TO2F73IZ6CPIA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429918137 (532h16m30.301001172s) [noise=true , address=145.79.227.28:4727 , piece_id=TYET6VIFJTNPG2S2MQKYQKJO6SHRQKIQDYNBIZBNVY2F467L4ZCA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.430614593 (532h16m30.301697628s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:32.134839415 (532h16m32.00592245s) [upload=466807]: Next returned nil segment Dec 27 19:45:32.134851387 (532h16m32.005934432s) [upload=466807]: Waiting for error group managing segments... Dec 27 19:45:32.413663857 (532h16m32.284746892s) [noise=true , address=145.79.227.31:4799 , piece_id=3HO72R76OZWWSFUYF622GB6OOZTFQSFJPBFPZDM4YYJAQP2PRQQA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.420589615 (532h16m32.29167265s) [noise=true , address=145.79.227.15:4876 , piece_id=E5Q2BC4MK3NJO5O3XJHO7SQKFLZXNNWPH2MEAS7U4NU4P3EC3SXA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.421963530 (532h16m32.293046565s) [noise=true , address=145.79.227.4:4187 , piece_id=IDQSOPGS436FNQ2MH5HFDUALWXEDVEPAV7TB2RXC7IUQHTQFWQYQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.427098208 (532h16m32.298181253s) [noise=true , address=145.79.227.38:4978 , piece_id=DKHGAP3ALG7DTM2CUHKPP3RGHHYO6IYW3Q46GUW4YUZBIC3GYMFQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.435325524 (532h16m32.306408569s) [noise=true , address=145.79.227.22:4510 , piece_id=T5TBMCQIFOY442EXOZT3XXIB46HJT2ISE5ILBGDI3JQDZ5R6GXIQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.438898999 (532h16m32.309982034s) [noise=true , address=145.79.227.28:4727 , piece_id=TYET6VIFJTNPG2S2MQKYQKJO6SHRQKIQDYNBIZBNVY2F467L4ZCA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.442651824 (532h16m32.313734859s) [noise=true , address=145.79.227.51:4411 , piece_id=7LTOU2XSG5OBBLCFMIWTGU6EHVDU43WQD43OVZUU5LBSCXZC7MYQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.445326046 (532h16m32.316409112s) [noise=true , address=145.79.227.5:4223 , piece_id=LXAK2RJRU6TE7IDTRFIV5CZSZHK5EVYZ6LRG4ZGJO45MRXLRBLYA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.445426828 (532h16m32.316509864s) [noise=true , address=145.79.227.10:4403 , piece_id=G7MJF4LCYZIOJZLEO7ERE3INJWJZ5JZ3S3GRIC52TNVJDXLUY4HA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.467291953 (532h16m32.338374988s) [noise=true , address=145.79.227.52:4446 , piece_id=ZI4I5HHEWVCDGHK5PIWMA55LTVRJQOYZR6RQWN7T3MZ4WIK4UX6A, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.606582909 (532h16m32.477665954s) [noise=true , address=77.90.31.13:4629 , piece_id=DLVRLXWLPV5YK4MU3NUYVXESMPK5DJHL46XMNZ3Q6EPPXW5N6TOA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.645386708 (532h16m32.516469743s) [noise=true , address=45.91.116.9:4262 , piece_id=J4RMOAJ3SUWATSDLVKEUJPTAWLOZATWKNU56YQ2TO2F73IZ6CPIA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.689000717 (532h16m32.560083752s) [noise=true , address=45.91.116.12:4227 , piece_id=LQTDZVZDXLCJCC7BT4MDIQY6O7C7EWXAXB57FYQHF7INIA2Q7MRA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.689019143 (532h16m32.560102178s) [seg_pos={0 0}, upload=466807]: Segment reached optimal threshold of 13 pieces. Dec 27 19:45:32.689225004 (532h16m32.560308039s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689232549 (532h16m32.560315594s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689486783 (532h16m32.560569828s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689494537 (532h16m32.560577572s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689663990 (532h16m32.560747035s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689672015 (532h16m32.56075505s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689689349 (532h16m32.560772394s) [seg_pos={0 0}, upload=466807]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:45:32.689885522 (532h16m32.560968557s) [upload=466807]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:45:32.689893658 (532h16m32.560976693s) [upload=466807]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:45:32.982897359 (532h16m32.853980394s) [upload=466807]: Done uploading segments... Dec 27 19:45:32.982917677 (532h16m32.854000723s) [upload=466807]: Upload finished. err: <nil> Duration: 5.844053992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........X....XXX..XX....X...XXXX...X..X..........X..X.X.X.X] duration: 30.003018162s trace: https://jaeger.dev.storj.io/trace/6cb8bf31d0386b6 FAIL (2025-12-27 19:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:45:30.180765025 (532h16m30.05184807s) [upload=466807]: Starting upload Dec 27 19:45:30.180854125 (532h16m30.05193716s) [upload=466807]: Done starting upload Dec 27 19:45:30.180899090 (532h16m30.051982125s) [upload=466807]: Uploading segments... Dec 27 19:45:30.181034428 (532h16m30.052117473s) [upload=466807]: Got next segment. Inline: false Dec 27 19:45:30.181065147 (532h16m30.052148192s) [upload=466807]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:45:30.181069295 (532h16m30.05215234s) [upload=466807]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:45:30.425047633 (532h16m30.296130668s) [seg_pos={0 0}, upload=466807]: Begin upload segment... Dec 27 19:45:30.425112035 (532h16m30.296195081s) [seg_pos={0 0}, upload=466807]: Done begin upload segment. Dec 27 19:45:30.425175927 (532h16m30.296258962s) [noise=true , address=145.79.227.4:4187 , piece_id=IDQSOPGS436FNQ2MH5HFDUALWXEDVEPAV7TB2RXC7IUQHTQFWQYQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425214080 (532h16m30.296297125s) [noise=true , address=77.90.31.13:4629 , piece_id=DLVRLXWLPV5YK4MU3NUYVXESMPK5DJHL46XMNZ3Q6EPPXW5N6TOA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425755762 (532h16m30.296838797s) [noise=true , address=145.79.227.15:4876 , piece_id=E5Q2BC4MK3NJO5O3XJHO7SQKFLZXNNWPH2MEAS7U4NU4P3EC3SXA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.425844220 (532h16m30.296927255s) [noise=true , address=145.79.227.51:4411 , piece_id=7LTOU2XSG5OBBLCFMIWTGU6EHVDU43WQD43OVZUU5LBSCXZC7MYQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.426224223 (532h16m30.297307268s) [noise=true , address=145.79.227.31:4799 , piece_id=3HO72R76OZWWSFUYF622GB6OOZTFQSFJPBFPZDM4YYJAQP2PRQQA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.426722041 (532h16m30.297805076s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427080665 (532h16m30.29816371s) [noise=true , address=145.79.227.38:4978 , piece_id=DKHGAP3ALG7DTM2CUHKPP3RGHHYO6IYW3Q46GUW4YUZBIC3GYMFQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427392148 (532h16m30.298475193s) [noise=true , address=145.79.227.5:4223 , piece_id=LXAK2RJRU6TE7IDTRFIV5CZSZHK5EVYZ6LRG4ZGJO45MRXLRBLYA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.427730662 (532h16m30.298813707s) [noise=true , address=45.91.116.12:4227 , piece_id=LQTDZVZDXLCJCC7BT4MDIQY6O7C7EWXAXB57FYQHF7INIA2Q7MRA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428123290 (532h16m30.299206335s) [noise=true , address=145.79.227.10:4403 , piece_id=G7MJF4LCYZIOJZLEO7ERE3INJWJZ5JZ3S3GRIC52TNVJDXLUY4HA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428610367 (532h16m30.299693412s) [noise=true , address=145.79.227.22:4510 , piece_id=T5TBMCQIFOY442EXOZT3XXIB46HJT2ISE5ILBGDI3JQDZ5R6GXIQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.428914897 (532h16m30.299997932s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429424056 (532h16m30.300507092s) [noise=true , address=145.79.227.52:4446 , piece_id=ZI4I5HHEWVCDGHK5PIWMA55LTVRJQOYZR6RQWN7T3MZ4WIK4UX6A, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429482578 (532h16m30.300565613s) [noise=true , address=45.91.116.9:4262 , piece_id=J4RMOAJ3SUWATSDLVKEUJPTAWLOZATWKNU56YQ2TO2F73IZ6CPIA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.429918137 (532h16m30.301001172s) [noise=true , address=145.79.227.28:4727 , piece_id=TYET6VIFJTNPG2S2MQKYQKJO6SHRQKIQDYNBIZBNVY2F467L4ZCA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:30.430614593 (532h16m30.301697628s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Uploading piece... Dec 27 19:45:32.134839415 (532h16m32.00592245s) [upload=466807]: Next returned nil segment Dec 27 19:45:32.134851387 (532h16m32.005934432s) [upload=466807]: Waiting for error group managing segments... Dec 27 19:45:32.413663857 (532h16m32.284746892s) [noise=true , address=145.79.227.31:4799 , piece_id=3HO72R76OZWWSFUYF622GB6OOZTFQSFJPBFPZDM4YYJAQP2PRQQA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.420589615 (532h16m32.29167265s) [noise=true , address=145.79.227.15:4876 , piece_id=E5Q2BC4MK3NJO5O3XJHO7SQKFLZXNNWPH2MEAS7U4NU4P3EC3SXA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.421963530 (532h16m32.293046565s) [noise=true , address=145.79.227.4:4187 , piece_id=IDQSOPGS436FNQ2MH5HFDUALWXEDVEPAV7TB2RXC7IUQHTQFWQYQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.427098208 (532h16m32.298181253s) [noise=true , address=145.79.227.38:4978 , piece_id=DKHGAP3ALG7DTM2CUHKPP3RGHHYO6IYW3Q46GUW4YUZBIC3GYMFQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.435325524 (532h16m32.306408569s) [noise=true , address=145.79.227.22:4510 , piece_id=T5TBMCQIFOY442EXOZT3XXIB46HJT2ISE5ILBGDI3JQDZ5R6GXIQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.438898999 (532h16m32.309982034s) [noise=true , address=145.79.227.28:4727 , piece_id=TYET6VIFJTNPG2S2MQKYQKJO6SHRQKIQDYNBIZBNVY2F467L4ZCA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.442651824 (532h16m32.313734859s) [noise=true , address=145.79.227.51:4411 , piece_id=7LTOU2XSG5OBBLCFMIWTGU6EHVDU43WQD43OVZUU5LBSCXZC7MYQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.445326046 (532h16m32.316409112s) [noise=true , address=145.79.227.5:4223 , piece_id=LXAK2RJRU6TE7IDTRFIV5CZSZHK5EVYZ6LRG4ZGJO45MRXLRBLYA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.445426828 (532h16m32.316509864s) [noise=true , address=145.79.227.10:4403 , piece_id=G7MJF4LCYZIOJZLEO7ERE3INJWJZ5JZ3S3GRIC52TNVJDXLUY4HA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.467291953 (532h16m32.338374988s) [noise=true , address=145.79.227.52:4446 , piece_id=ZI4I5HHEWVCDGHK5PIWMA55LTVRJQOYZR6RQWN7T3MZ4WIK4UX6A, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.606582909 (532h16m32.477665954s) [noise=true , address=77.90.31.13:4629 , piece_id=DLVRLXWLPV5YK4MU3NUYVXESMPK5DJHL46XMNZ3Q6EPPXW5N6TOA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.645386708 (532h16m32.516469743s) [noise=true , address=45.91.116.9:4262 , piece_id=J4RMOAJ3SUWATSDLVKEUJPTAWLOZATWKNU56YQ2TO2F73IZ6CPIA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.689000717 (532h16m32.560083752s) [noise=true , address=45.91.116.12:4227 , piece_id=LQTDZVZDXLCJCC7BT4MDIQY6O7C7EWXAXB57FYQHF7INIA2Q7MRA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: <nil> Dec 27 19:45:32.689019143 (532h16m32.560102178s) [seg_pos={0 0}, upload=466807]: Segment reached optimal threshold of 13 pieces. Dec 27 19:45:32.689225004 (532h16m32.560308039s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689232549 (532h16m32.560315594s) [noise=true , address=77.90.31.14:4664 , piece_id=ENRBVRXMQ4WGACJO53HPHCYGHBEITTPRA4O6UEUNJQLEAGFFT5OA, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689486783 (532h16m32.560569828s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689494537 (532h16m32.560577572s) [noise=true , address=45.91.116.14:4191 , piece_id=N2ZO5SONB7TN77QHFA33RKMCYI7CD2DWF75SHRJF5XZYZ5IQEEWA, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689663990 (532h16m32.560747035s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 27 19:45:32.689672015 (532h16m32.56075505s) [noise=true , address=77.90.31.16:4735 , piece_id=QBSQ46OEYQ7KRKLNR3O7FV4OVLFNZQ2B4TRA7NKPOHD4RHKOCPTQ, seg_pos={0 0}, upload=466807]: Long tail cancellation due to optimal threshold Dec 27 19:45:32.689689349 (532h16m32.560772394s) [seg_pos={0 0}, upload=466807]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:45:32.689885522 (532h16m32.560968557s) [upload=466807]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:45:32.689893658 (532h16m32.560976693s) [upload=466807]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:45:32.982897359 (532h16m32.853980394s) [upload=466807]: Done uploading segments... Dec 27 19:45:32.982917677 (532h16m32.854000723s) [upload=466807]: Upload finished. err: <nil> Duration: 30.003045254s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.425938723s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:45:44 UTC) orbiter warning: context deadline exceeded Duration: 3.425968149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.607583527s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:44:58 UTC) orbiter warning: context deadline exceeded Duration: 3.607609898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.........X....XXX..XX....X...XXXX...X..X..........X..X.X.X.] duration: 6.059859401s trace: https://jaeger.dev.storj.io/trace/1e6f960701c74b97 FAIL (2025-12-27 19:44:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:43:38.314312626 (532h14m38.185395671s) [upload=466778]: Starting upload Dec 27 19:43:38.314398690 (532h14m38.185481725s) [upload=466778]: Done starting upload Dec 27 19:43:38.314531542 (532h14m38.185614577s) [upload=466778]: Uploading segments... Dec 27 19:43:38.314651621 (532h14m38.185734656s) [upload=466778]: Got next segment. Inline: false Dec 27 19:43:38.314718769 (532h14m38.185801804s) [upload=466778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:43:38.314723348 (532h14m38.185806383s) [upload=466778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:43:38.559886595 (532h14m38.43096964s) [seg_pos={0 0}, upload=466778]: Begin upload segment... Dec 27 19:43:38.559938505 (532h14m38.43102155s) [seg_pos={0 0}, upload=466778]: Done begin upload segment. Dec 27 19:43:38.559971357 (532h14m38.431054392s) [noise=true , address=77.90.31.10:4517 , piece_id=6M7EZPNK37KAF277HZPJU7HRC53K5WXIY43CPAOQEET4VUQX4W3Q, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.561935206 (532h14m38.433018251s) [noise=true , address=45.91.116.13:4153 , piece_id=D5FUVDT6N52IKIUG7LK33G4TR3DSKYHW5AFXUKK7BU4VASNACWOQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.562753775 (532h14m38.43383681s) [noise=true , address=145.79.227.12:4765 , piece_id=NKZKCGIRKEQGU4M7USA4RSG36X2W3F7AI4AEMUWPAEEONSJ5KZWQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563128770 (532h14m38.434211805s) [noise=true , address=45.91.116.10:4081 , piece_id=JW5JNT4C2QSTUH3SIS4K42KEKGFCWVIHOWN4AVQW2SMVI4NTU2EA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563158966 (532h14m38.434242011s) [noise=true , address=45.91.116.7:4011 , piece_id=TTQAE4SUAVFZRNOSUUXQTP7WUHNW56B43I4IY75OR22F3JPQRV5A, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563479087 (532h14m38.434562132s) [noise=true , address=145.79.227.46:4603 , piece_id=T2OR5NTSPLZ2CLDYFCY3EF7GBLFBFCHAMDI6HSG4ZGFOXZPXVUFQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563623822 (532h14m38.434706857s) [noise=true , address=145.79.227.4:4187 , piece_id=LXU6IB5GTIWML4JLXXN3YN4OLQYYO2IFV3V6K5VUIBPDCE2323OA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563926339 (532h14m38.435009374s) [noise=true , address=145.79.227.22:4510 , piece_id=QWNQZHBKCVAIBYM4O2FFDJPQLGWELWQ6FS5BPF6QVZYG3T4PJ2OA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.564063469 (532h14m38.435146504s) [noise=true , address=145.79.227.48:4303 , piece_id=GFO36OK3BYUOEC5NQV5GHNLZ4NADJX4G3XHEPFD52HFXTMKDDWIA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.564399108 (532h14m38.435482153s) [noise=true , address=45.91.116.14:4189 , piece_id=C2EJYXCHRDJZ7EWX4WM7C42JXC2SYI762FB33QMQI55NXGP2RNAQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.564551418 (532h14m38.435634463s) [noise=true , address=145.79.227.42:4121 , piece_id=ZANLJJ4MG6DDRFSQTCLENYEZXL4RUNHDHWJTC4TC24CXBRV4U45Q, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.564843805 (532h14m38.43592684s) [noise=true , address=77.90.31.17:4773 , piece_id=FATFSDBRQWWTERLNBLX6QCGFA4OUWCTZO7HM7DHXOZ3LMMYJJZAQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.565038585 (532h14m38.436121631s) [noise=true , address=145.79.227.51:4409 , piece_id=YGINY5MOHKLH433YB2XZ562QJQHOPLTPFZD5VLHMOCVAFFWBA6XQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.565264186 (532h14m38.436347231s) [noise=true , address=145.79.227.32:4836 , piece_id=UNST44CJ3A7Y2SMLGUOOPWDOEZDFH27AIPMZ3WMUV2AEZZQC2BFQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.565518860 (532h14m38.436601895s) [noise=true , address=145.79.227.21:4474 , piece_id=7GUBIYCOGX2F5GYPV4TN7EC3DX5RU3IYTK6CJQVBEKLCEQ4DMANA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.566237960 (532h14m38.437321005s) [noise=true , address=145.79.227.25:4619 , piece_id=ZPXXBE27NAJKYRUUBS6PXHFMHNJYHIEF3PUCWAWXYJ4N43C2EPWQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:40.287308915 (532h14m40.15839195s) [upload=466778]: Next returned nil segment Dec 27 19:43:40.287327401 (532h14m40.158410446s) [upload=466778]: Waiting for error group managing segments... Dec 27 19:43:40.577781939 (532h14m40.448864974s) [noise=true , address=145.79.227.25:4619 , piece_id=ZPXXBE27NAJKYRUUBS6PXHFMHNJYHIEF3PUCWAWXYJ4N43C2EPWQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.578633249 (532h14m40.449716284s) [noise=true , address=145.79.227.48:4303 , piece_id=GFO36OK3BYUOEC5NQV5GHNLZ4NADJX4G3XHEPFD52HFXTMKDDWIA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.588153748 (532h14m40.459236783s) [noise=true , address=145.79.227.51:4409 , piece_id=YGINY5MOHKLH433YB2XZ562QJQHOPLTPFZD5VLHMOCVAFFWBA6XQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.589775716 (532h14m40.460858761s) [noise=true , address=145.79.227.4:4187 , piece_id=LXU6IB5GTIWML4JLXXN3YN4OLQYYO2IFV3V6K5VUIBPDCE2323OA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.591777108 (532h14m40.462860143s) [noise=true , address=145.79.227.32:4836 , piece_id=UNST44CJ3A7Y2SMLGUOOPWDOEZDFH27AIPMZ3WMUV2AEZZQC2BFQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.592478684 (532h14m40.463561719s) [noise=true , address=145.79.227.46:4603 , piece_id=T2OR5NTSPLZ2CLDYFCY3EF7GBLFBFCHAMDI6HSG4ZGFOXZPXVUFQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.603990224 (532h14m40.475073259s) [noise=true , address=145.79.227.22:4510 , piece_id=QWNQZHBKCVAIBYM4O2FFDJPQLGWELWQ6FS5BPF6QVZYG3T4PJ2OA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.609662465 (532h14m40.48074551s) [noise=true , address=145.79.227.12:4765 , piece_id=NKZKCGIRKEQGU4M7USA4RSG36X2W3F7AI4AEMUWPAEEONSJ5KZWQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.622267076 (532h14m40.493350121s) [noise=true , address=145.79.227.42:4121 , piece_id=ZANLJJ4MG6DDRFSQTCLENYEZXL4RUNHDHWJTC4TC24CXBRV4U45Q, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.624370832 (532h14m40.495453867s) [noise=true , address=145.79.227.21:4474 , piece_id=7GUBIYCOGX2F5GYPV4TN7EC3DX5RU3IYTK6CJQVBEKLCEQ4DMANA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.775171023 (532h14m40.646254068s) [noise=true , address=77.90.31.17:4773 , piece_id=FATFSDBRQWWTERLNBLX6QCGFA4OUWCTZO7HM7DHXOZ3LMMYJJZAQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.835942803 (532h14m40.707025848s) [noise=true , address=45.91.116.13:4153 , piece_id=D5FUVDT6N52IKIUG7LK33G4TR3DSKYHW5AFXUKK7BU4VASNACWOQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.859511522 (532h14m40.730594567s) [noise=true , address=77.90.31.10:4517 , piece_id=6M7EZPNK37KAF277HZPJU7HRC53K5WXIY43CPAOQEET4VUQX4W3Q, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.859533094 (532h14m40.730616159s) [seg_pos={0 0}, upload=466778]: Segment reached optimal threshold of 13 pieces. Dec 27 19:43:40.859841400 (532h14m40.730924445s) [noise=true , address=45.91.116.14:4189 , piece_id=C2EJYXCHRDJZ7EWX4WM7C42JXC2SYI762FB33QMQI55NXGP2RNAQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 19:43:40.859851710 (532h14m40.730934755s) [noise=true , address=45.91.116.14:4189 , piece_id=C2EJYXCHRDJZ7EWX4WM7C42JXC2SYI762FB33QMQI55NXGP2RNAQ, seg_pos={0 0}, upload=466778]: Long tail cancellation due to optimal threshold Dec 27 19:43:40.860099432 (532h14m40.731182477s) [noise=true , address=45.91.116.10:4081 , piece_id=JW5JNT4C2QSTUH3SIS4K42KEKGFCWVIHOWN4AVQW2SMVI4NTU2EA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 19:43:40.860108188 (532h14m40.731191233s) [noise=true , address=45.91.116.10:4081 , piece_id=JW5JNT4C2QSTUH3SIS4K42KEKGFCWVIHOWN4AVQW2SMVI4NTU2EA, seg_pos={0 0}, upload=466778]: Long tail cancellation due to optimal threshold Dec 27 19:43:40.860188341 (532h14m40.731271376s) [noise=true , address=45.91.116.7:4011 , piece_id=TTQAE4SUAVFZRNOSUUXQTP7WUHNW56B43I4IY75OR22F3JPQRV5A, seg_pos={0 0}, upload=466778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 19:43:40.860226334 (532h14m40.731309369s) [noise=true , address=45.91.116.7:4011 , piece_id=TTQAE4SUAVFZRNOSUUXQTP7WUHNW56B43I4IY75OR22F3JPQRV5A, seg_pos={0 0}, upload=466778]: Long tail cancellation due to optimal threshold Dec 27 19:43:40.860250980 (532h14m40.731334015s) [seg_pos={0 0}, upload=466778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:43:40.860352844 (532h14m40.731435879s) [upload=466778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:43:40.860359587 (532h14m40.731442632s) [upload=466778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:43:41.149574445 (532h14m41.02065749s) [upload=466778]: Done uploading segments... Dec 27 19:43:41.149595314 (532h14m41.020678349s) [upload=466778]: Upload finished. err: <nil> Duration: 6.059885381s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.580523402s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:44:29 UTC) orbiter warning: context deadline exceeded Duration: 3.580599388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.........X....XXX..XX....X...XXXX...X..X..........X..X.X.X] duration: 30.002776816s trace: https://jaeger.dev.storj.io/trace/1e6f960701c74b97 FAIL (2025-12-27 19:44:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:43:38.314312626 (532h14m38.185395671s) [upload=466778]: Starting upload Dec 27 19:43:38.314398690 (532h14m38.185481725s) [upload=466778]: Done starting upload Dec 27 19:43:38.314531542 (532h14m38.185614577s) [upload=466778]: Uploading segments... Dec 27 19:43:38.314651621 (532h14m38.185734656s) [upload=466778]: Got next segment. Inline: false Dec 27 19:43:38.314718769 (532h14m38.185801804s) [upload=466778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:43:38.314723348 (532h14m38.185806383s) [upload=466778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:43:38.559886595 (532h14m38.43096964s) [seg_pos={0 0}, upload=466778]: Begin upload segment... Dec 27 19:43:38.559938505 (532h14m38.43102155s) [seg_pos={0 0}, upload=466778]: Done begin upload segment. Dec 27 19:43:38.559971357 (532h14m38.431054392s) [noise=true , address=77.90.31.10:4517 , piece_id=6M7EZPNK37KAF277HZPJU7HRC53K5WXIY43CPAOQEET4VUQX4W3Q, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.561935206 (532h14m38.433018251s) [noise=true , address=45.91.116.13:4153 , piece_id=D5FUVDT6N52IKIUG7LK33G4TR3DSKYHW5AFXUKK7BU4VASNACWOQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.562753775 (532h14m38.43383681s) [noise=true , address=145.79.227.12:4765 , piece_id=NKZKCGIRKEQGU4M7USA4RSG36X2W3F7AI4AEMUWPAEEONSJ5KZWQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563128770 (532h14m38.434211805s) [noise=true , address=45.91.116.10:4081 , piece_id=JW5JNT4C2QSTUH3SIS4K42KEKGFCWVIHOWN4AVQW2SMVI4NTU2EA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563158966 (532h14m38.434242011s) [noise=true , address=45.91.116.7:4011 , piece_id=TTQAE4SUAVFZRNOSUUXQTP7WUHNW56B43I4IY75OR22F3JPQRV5A, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563479087 (532h14m38.434562132s) [noise=true , address=145.79.227.46:4603 , piece_id=T2OR5NTSPLZ2CLDYFCY3EF7GBLFBFCHAMDI6HSG4ZGFOXZPXVUFQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563623822 (532h14m38.434706857s) [noise=true , address=145.79.227.4:4187 , piece_id=LXU6IB5GTIWML4JLXXN3YN4OLQYYO2IFV3V6K5VUIBPDCE2323OA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.563926339 (532h14m38.435009374s) [noise=true , address=145.79.227.22:4510 , piece_id=QWNQZHBKCVAIBYM4O2FFDJPQLGWELWQ6FS5BPF6QVZYG3T4PJ2OA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.564063469 (532h14m38.435146504s) [noise=true , address=145.79.227.48:4303 , piece_id=GFO36OK3BYUOEC5NQV5GHNLZ4NADJX4G3XHEPFD52HFXTMKDDWIA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.564399108 (532h14m38.435482153s) [noise=true , address=45.91.116.14:4189 , piece_id=C2EJYXCHRDJZ7EWX4WM7C42JXC2SYI762FB33QMQI55NXGP2RNAQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.564551418 (532h14m38.435634463s) [noise=true , address=145.79.227.42:4121 , piece_id=ZANLJJ4MG6DDRFSQTCLENYEZXL4RUNHDHWJTC4TC24CXBRV4U45Q, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.564843805 (532h14m38.43592684s) [noise=true , address=77.90.31.17:4773 , piece_id=FATFSDBRQWWTERLNBLX6QCGFA4OUWCTZO7HM7DHXOZ3LMMYJJZAQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.565038585 (532h14m38.436121631s) [noise=true , address=145.79.227.51:4409 , piece_id=YGINY5MOHKLH433YB2XZ562QJQHOPLTPFZD5VLHMOCVAFFWBA6XQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.565264186 (532h14m38.436347231s) [noise=true , address=145.79.227.32:4836 , piece_id=UNST44CJ3A7Y2SMLGUOOPWDOEZDFH27AIPMZ3WMUV2AEZZQC2BFQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.565518860 (532h14m38.436601895s) [noise=true , address=145.79.227.21:4474 , piece_id=7GUBIYCOGX2F5GYPV4TN7EC3DX5RU3IYTK6CJQVBEKLCEQ4DMANA, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:38.566237960 (532h14m38.437321005s) [noise=true , address=145.79.227.25:4619 , piece_id=ZPXXBE27NAJKYRUUBS6PXHFMHNJYHIEF3PUCWAWXYJ4N43C2EPWQ, seg_pos={0 0}, upload=466778]: Uploading piece... Dec 27 19:43:40.287308915 (532h14m40.15839195s) [upload=466778]: Next returned nil segment Dec 27 19:43:40.287327401 (532h14m40.158410446s) [upload=466778]: Waiting for error group managing segments... Dec 27 19:43:40.577781939 (532h14m40.448864974s) [noise=true , address=145.79.227.25:4619 , piece_id=ZPXXBE27NAJKYRUUBS6PXHFMHNJYHIEF3PUCWAWXYJ4N43C2EPWQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.578633249 (532h14m40.449716284s) [noise=true , address=145.79.227.48:4303 , piece_id=GFO36OK3BYUOEC5NQV5GHNLZ4NADJX4G3XHEPFD52HFXTMKDDWIA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.588153748 (532h14m40.459236783s) [noise=true , address=145.79.227.51:4409 , piece_id=YGINY5MOHKLH433YB2XZ562QJQHOPLTPFZD5VLHMOCVAFFWBA6XQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.589775716 (532h14m40.460858761s) [noise=true , address=145.79.227.4:4187 , piece_id=LXU6IB5GTIWML4JLXXN3YN4OLQYYO2IFV3V6K5VUIBPDCE2323OA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.591777108 (532h14m40.462860143s) [noise=true , address=145.79.227.32:4836 , piece_id=UNST44CJ3A7Y2SMLGUOOPWDOEZDFH27AIPMZ3WMUV2AEZZQC2BFQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.592478684 (532h14m40.463561719s) [noise=true , address=145.79.227.46:4603 , piece_id=T2OR5NTSPLZ2CLDYFCY3EF7GBLFBFCHAMDI6HSG4ZGFOXZPXVUFQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.603990224 (532h14m40.475073259s) [noise=true , address=145.79.227.22:4510 , piece_id=QWNQZHBKCVAIBYM4O2FFDJPQLGWELWQ6FS5BPF6QVZYG3T4PJ2OA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.609662465 (532h14m40.48074551s) [noise=true , address=145.79.227.12:4765 , piece_id=NKZKCGIRKEQGU4M7USA4RSG36X2W3F7AI4AEMUWPAEEONSJ5KZWQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.622267076 (532h14m40.493350121s) [noise=true , address=145.79.227.42:4121 , piece_id=ZANLJJ4MG6DDRFSQTCLENYEZXL4RUNHDHWJTC4TC24CXBRV4U45Q, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.624370832 (532h14m40.495453867s) [noise=true , address=145.79.227.21:4474 , piece_id=7GUBIYCOGX2F5GYPV4TN7EC3DX5RU3IYTK6CJQVBEKLCEQ4DMANA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.775171023 (532h14m40.646254068s) [noise=true , address=77.90.31.17:4773 , piece_id=FATFSDBRQWWTERLNBLX6QCGFA4OUWCTZO7HM7DHXOZ3LMMYJJZAQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.835942803 (532h14m40.707025848s) [noise=true , address=45.91.116.13:4153 , piece_id=D5FUVDT6N52IKIUG7LK33G4TR3DSKYHW5AFXUKK7BU4VASNACWOQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.859511522 (532h14m40.730594567s) [noise=true , address=77.90.31.10:4517 , piece_id=6M7EZPNK37KAF277HZPJU7HRC53K5WXIY43CPAOQEET4VUQX4W3Q, seg_pos={0 0}, upload=466778]: Done uploading piece. err: <nil> Dec 27 19:43:40.859533094 (532h14m40.730616159s) [seg_pos={0 0}, upload=466778]: Segment reached optimal threshold of 13 pieces. Dec 27 19:43:40.859841400 (532h14m40.730924445s) [noise=true , address=45.91.116.14:4189 , piece_id=C2EJYXCHRDJZ7EWX4WM7C42JXC2SYI762FB33QMQI55NXGP2RNAQ, seg_pos={0 0}, upload=466778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 19:43:40.859851710 (532h14m40.730934755s) [noise=true , address=45.91.116.14:4189 , piece_id=C2EJYXCHRDJZ7EWX4WM7C42JXC2SYI762FB33QMQI55NXGP2RNAQ, seg_pos={0 0}, upload=466778]: Long tail cancellation due to optimal threshold Dec 27 19:43:40.860099432 (532h14m40.731182477s) [noise=true , address=45.91.116.10:4081 , piece_id=JW5JNT4C2QSTUH3SIS4K42KEKGFCWVIHOWN4AVQW2SMVI4NTU2EA, seg_pos={0 0}, upload=466778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 19:43:40.860108188 (532h14m40.731191233s) [noise=true , address=45.91.116.10:4081 , piece_id=JW5JNT4C2QSTUH3SIS4K42KEKGFCWVIHOWN4AVQW2SMVI4NTU2EA, seg_pos={0 0}, upload=466778]: Long tail cancellation due to optimal threshold Dec 27 19:43:40.860188341 (532h14m40.731271376s) [noise=true , address=45.91.116.7:4011 , piece_id=TTQAE4SUAVFZRNOSUUXQTP7WUHNW56B43I4IY75OR22F3JPQRV5A, seg_pos={0 0}, upload=466778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 19:43:40.860226334 (532h14m40.731309369s) [noise=true , address=45.91.116.7:4011 , piece_id=TTQAE4SUAVFZRNOSUUXQTP7WUHNW56B43I4IY75OR22F3JPQRV5A, seg_pos={0 0}, upload=466778]: Long tail cancellation due to optimal threshold Dec 27 19:43:40.860250980 (532h14m40.731334015s) [seg_pos={0 0}, upload=466778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:43:40.860352844 (532h14m40.731435879s) [upload=466778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:43:40.860359587 (532h14m40.731442632s) [upload=466778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:43:41.149574445 (532h14m41.02065749s) [upload=466778]: Done uploading segments... Dec 27 19:43:41.149595314 (532h14m41.020678349s) [upload=466778]: Upload finished. err: <nil> Duration: 30.002801934s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.534767172s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:43:36 UTC) orbiter warning: context deadline exceeded Duration: 3.534792211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.........X....XXX..XX....X...XXXX...X..X..........X..X.X.] duration: 5.904752801s trace: https://jaeger.dev.storj.io/trace/565fd00194fdeee FAIL (2025-12-27 19:43:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: TEMDFCEMWM4AVEHFNHXZXMFZ232EMGXPESVDV4HIIHT4ASGKDKJA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GUE4PD4H222CDIGX7EPTDN2HR5EDWAJGP4IHX4VCZKN6MWTW4W2Q) context deadline exceeded ======LOGS====== :42:15.179119183 (532h13m15.050202218s) [upload=466760]: Starting upload Dec 27 19:42:15.179191671 (532h13m15.050274706s) [upload=466760]: Done starting upload Dec 27 19:42:15.179275530 (532h13m15.050358575s) [upload=466760]: Uploading segments... Dec 27 19:42:15.179380900 (532h13m15.050463946s) [upload=466760]: Got next segment. Inline: false Dec 27 19:42:15.179419083 (532h13m15.050502118s) [upload=466760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:42:15.179424324 (532h13m15.050507359s) [upload=466760]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:42:15.424178209 (532h13m15.295261254s) [seg_pos={0 0}, upload=466760]: Begin upload segment... Dec 27 19:42:15.424302015 (532h13m15.29538505s) [seg_pos={0 0}, upload=466760]: Done begin upload segment. Dec 27 19:42:15.424330299 (532h13m15.295413334s) [noise=true , address=145.79.227.22:4510 , piece_id=EW7GAA3RVZDKKPRVXBYYLXOBTHHPSNHVCIUPWVXCCVBDXJP6BN7A, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.424450617 (532h13m15.295533663s) [noise=true , address=145.79.227.46:4602 , piece_id=ODRK3VGLSJSKDAMGAEKOGEHAXQOS7TPFJOI2LKVOPARJMPGG435A, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.424968143 (532h13m15.296051188s) [noise=true , address=145.79.227.20:4438 , piece_id=RJDI5FQDC4IMYCATOJPKORLUNHO7NIDU6ZSRZCP5HRLWOOQHHLSA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.425755723 (532h13m15.296838758s) [noise=true , address=45.91.116.12:4226 , piece_id=FDTTRHXECLDRCGRIBOISE25D34SKPQ4MUGI2AZ3QHXEXCU4MZT5Q, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.427137795 (532h13m15.29822083s) [noise=true , address=45.91.116.5:4300 , piece_id=GUE4PD4H222CDIGX7EPTDN2HR5EDWAJGP4IHX4VCZKN6MWTW4W2Q, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.428124243 (532h13m15.299207288s) [noise=true , address=145.79.227.31:4799 , piece_id=FRYDBCIBVM6ABHTMJJORMVZC736K3RIINWX7VPNVN5Q2HCSKR53A, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.428583698 (532h13m15.299666733s) [noise=true , address=145.79.227.45:4231 , piece_id=WNQUP6IGJ7YOX42OXKPW2SXS7BS4LOX3252LI5TXFBSLGA4UJGYQ, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.429228616 (532h13m15.300311671s) [noise=true , address=145.79.227.37:4565 , piece_id=ACOXUNZEBULHFJKCECKNH53HVJCRALTXWJTDVXMPDYSHEP3WI7ZA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.429956953 (532h13m15.301040018s) [noise=true , address=145.79.227.17:4971 , piece_id=MCU5MAA3TNQ2JQJPI7COBKWDCIMRTXYGHOZVUOFMCM2FC76IK2YQ, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.431262489 (532h13m15.302345574s) [noise=true , address=145.79.227.47:4267 , piece_id=B5RVDJJW7SQSUAFFMOGLILQIWIKVKXZ23ADIZH6OPR2TPCDYHXHA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.432796540 (532h13m15.303879575s) [noise=true , address=145.79.227.30:4761 , piece_id=Y2ECXZUBP74LZU25U54LV57YYWGGVDNQ42BMLT62M7HCLQZGJQWA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.434762945 (532h13m15.30584624s) [noise=true , address=145.79.227.11:4731 , piece_id=V7MENKXI2AAPYVMMIRBCSHCR6ULPMJ7GFFI47OQVBQUOZYTMHU3A, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.439800126 (532h13m15.310883181s) [noise=true , address=145.79.227.51:4410 , piece_id=SAIOGZES5YWRJFKT6MER7CJ4JF4CL2RM2BFSHPD53HWFRTHOOZAQ, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.441923539 (532h13m15.313006584s) [noise=true , address=145.79.227.9:4365 , piece_id=K2NSPPIYWYYGGIU5MXXOYYMEAAWJJOT27RBKK2RDDDKQNRTSK54Q, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.444165699 (532h13m15.315248734s) [noise=true , address=77.90.31.13:4629 , piece_id=TEMDFCEMWM4AVEHFNHXZXMFZ232EMGXPESVDV4HIIHT4ASGKDKJA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.446168362 (532h13m15.317251497s) [noise=true , address=145.79.227.10:4403 , piece_id=D2U6KLTGKZFXTWTM4346RBBAISWW6NXBK6SDKWME5REAOQI7XRJA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:17.039841032 (532h13m16.910924067s) [upload=466760]: Next returned nil segment Dec 27 19:42:17.039858514 (532h13m16.91094155s) [upload=466760]: Waiting for error group managing segments... Dec 27 19:42:17.421723725 (532h13m17.29280676s) [noise=true , address=145.79.227.46:4602 , piece_id=ODRK3VGLSJSKDAMGAEKOGEHAXQOS7TPFJOI2LKVOPARJMPGG435A, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.443821032 (532h13m17.314904077s) [noise=true , address=145.79.227.20:4438 , piece_id=RJDI5FQDC4IMYCATOJPKORLUNHO7NIDU6ZSRZCP5HRLWOOQHHLSA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.444623550 (532h13m17.315706595s) [noise=true , address=145.79.227.10:4403 , piece_id=D2U6KLTGKZFXTWTM4346RBBAISWW6NXBK6SDKWME5REAOQI7XRJA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.446620302 (532h13m17.317703337s) [noise=true , address=145.79.227.37:4565 , piece_id=ACOXUNZEBULHFJKCECKNH53HVJCRALTXWJTDVXMPDYSHEP3WI7ZA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.450253079 (532h13m17.321336124s) [noise=true , address=145.79.227.51:4410 , piece_id=SAIOGZES5YWRJFKT6MER7CJ4JF4CL2RM2BFSHPD53HWFRTHOOZAQ, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.450472247 (532h13m17.321555282s) [noise=true , address=145.79.227.31:4799 , piece_id=FRYDBCIBVM6ABHTMJJORMVZC736K3RIINWX7VPNVN5Q2HCSKR53A, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.450774011 (532h13m17.321857046s) [noise=true , address=145.79.227.47:4267 , piece_id=B5RVDJJW7SQSUAFFMOGLILQIWIKVKXZ23ADIZH6OPR2TPCDYHXHA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.455679783 (532h13m17.326762818s) [noise=true , address=145.79.227.11:4731 , piece_id=V7MENKXI2AAPYVMMIRBCSHCR6ULPMJ7GFFI47OQVBQUOZYTMHU3A, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.459253097 (532h13m17.330336132s) [noise=true , address=145.79.227.22:4510 , piece_id=EW7GAA3RVZDKKPRVXBYYLXOBTHHPSNHVCIUPWVXCCVBDXJP6BN7A, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.462103215 (532h13m17.33318625s) [noise=true , address=145.79.227.45:4231 , piece_id=WNQUP6IGJ7YOX42OXKPW2SXS7BS4LOX3252LI5TXFBSLGA4UJGYQ, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.475778114 (532h13m17.346861149s) [noise=true , address=145.79.227.9:4365 , piece_id=K2NSPPIYWYYGGIU5MXXOYYMEAAWJJOT27RBKK2RDDDKQNRTSK54Q, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.678570046 (532h13m17.549653081s) [noise=true , address=45.91.116.5:4300 , piece_id=GUE4PD4H222CDIGX7EPTDN2HR5EDWAJGP4IHX4VCZKN6MWTW4W2Q, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.692586176 (532h13m17.563669211s) [noise=true , address=77.90.31.13:4629 , piece_id=TEMDFCEMWM4AVEHFNHXZXMFZ232EMGXPESVDV4HIIHT4ASGKDKJA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.692611213 (532h13m17.563694278s) [seg_pos={0 0}, upload=466760]: Segment reached optimal threshold of 13 pieces. Dec 27 19:42:17.692797358 (532h13m17.563880393s) [noise=true , address=145.79.227.17:4971 , piece_id=MCU5MAA3TNQ2JQJPI7COBKWDCIMRTXYGHOZVUOFMCM2FC76IK2YQ, seg_pos={0 0}, upload=466760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:44422->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.17:4971: use of closed network connection Dec 27 19:42:17.692803690 (532h13m17.563886725s) [noise=true , address=145.79.227.17:4971 , piece_id=MCU5MAA3TNQ2JQJPI7COBKWDCIMRTXYGHOZVUOFMCM2FC76IK2YQ, seg_pos={0 0}, upload=466760]: Long tail cancellation due to optimal threshold Dec 27 19:42:17.693008400 (532h13m17.564091445s) [noise=true , address=145.79.227.30:4761 , piece_id=Y2ECXZUBP74LZU25U54LV57YYWGGVDNQ42BMLT62M7HCLQZGJQWA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 19:42:17.693014311 (532h13m17.564097346s) [noise=true , address=145.79.227.30:4761 , piece_id=Y2ECXZUBP74LZU25U54LV57YYWGGVDNQ42BMLT62M7HCLQZGJQWA, seg_pos={0 0}, upload=466760]: Long tail cancellation due to optimal threshold Dec 27 19:42:17.693114512 (532h13m17.564197557s) [noise=true , address=45.91.116.12:4226 , piece_id=FDTTRHXECLDRCGRIBOISE25D34SKPQ4MUGI2AZ3QHXEXCU4MZT5Q, seg_pos={0 0}, upload=466760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 19:42:17.693120313 (532h13m17.564203348s) [noise=true , address=45.91.116.12:4226 , piece_id=FDTTRHXECLDRCGRIBOISE25D34SKPQ4MUGI2AZ3QHXEXCU4MZT5Q, seg_pos={0 0}, upload=466760]: Long tail cancellation due to optimal threshold Dec 27 19:42:17.693138026 (532h13m17.564221061s) [seg_pos={0 0}, upload=466760]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:42:17.693206206 (532h13m17.564289251s) [upload=466760]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:42:17.693211196 (532h13m17.564294241s) [upload=466760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:42:17.982000192 (532h13m17.853083227s) [upload=466760]: Done uploading segments... Dec 27 19:42:17.982019458 (532h13m17.853102493s) [upload=466760]: Upload finished. err: <nil> Duration: 5.904801554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.........X....XXX..XX....X...XXXX...X..X..........X..X.X] duration: 30.002768901s trace: https://jaeger.dev.storj.io/trace/565fd00194fdeee FAIL (2025-12-27 19:42:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: TEMDFCEMWM4AVEHFNHXZXMFZ232EMGXPESVDV4HIIHT4ASGKDKJA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GUE4PD4H222CDIGX7EPTDN2HR5EDWAJGP4IHX4VCZKN6MWTW4W2Q) context deadline exceeded ======LOGS====== :42:15.179119183 (532h13m15.050202218s) [upload=466760]: Starting upload Dec 27 19:42:15.179191671 (532h13m15.050274706s) [upload=466760]: Done starting upload Dec 27 19:42:15.179275530 (532h13m15.050358575s) [upload=466760]: Uploading segments... Dec 27 19:42:15.179380900 (532h13m15.050463946s) [upload=466760]: Got next segment. Inline: false Dec 27 19:42:15.179419083 (532h13m15.050502118s) [upload=466760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:42:15.179424324 (532h13m15.050507359s) [upload=466760]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:42:15.424178209 (532h13m15.295261254s) [seg_pos={0 0}, upload=466760]: Begin upload segment... Dec 27 19:42:15.424302015 (532h13m15.29538505s) [seg_pos={0 0}, upload=466760]: Done begin upload segment. Dec 27 19:42:15.424330299 (532h13m15.295413334s) [noise=true , address=145.79.227.22:4510 , piece_id=EW7GAA3RVZDKKPRVXBYYLXOBTHHPSNHVCIUPWVXCCVBDXJP6BN7A, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.424450617 (532h13m15.295533663s) [noise=true , address=145.79.227.46:4602 , piece_id=ODRK3VGLSJSKDAMGAEKOGEHAXQOS7TPFJOI2LKVOPARJMPGG435A, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.424968143 (532h13m15.296051188s) [noise=true , address=145.79.227.20:4438 , piece_id=RJDI5FQDC4IMYCATOJPKORLUNHO7NIDU6ZSRZCP5HRLWOOQHHLSA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.425755723 (532h13m15.296838758s) [noise=true , address=45.91.116.12:4226 , piece_id=FDTTRHXECLDRCGRIBOISE25D34SKPQ4MUGI2AZ3QHXEXCU4MZT5Q, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.427137795 (532h13m15.29822083s) [noise=true , address=45.91.116.5:4300 , piece_id=GUE4PD4H222CDIGX7EPTDN2HR5EDWAJGP4IHX4VCZKN6MWTW4W2Q, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.428124243 (532h13m15.299207288s) [noise=true , address=145.79.227.31:4799 , piece_id=FRYDBCIBVM6ABHTMJJORMVZC736K3RIINWX7VPNVN5Q2HCSKR53A, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.428583698 (532h13m15.299666733s) [noise=true , address=145.79.227.45:4231 , piece_id=WNQUP6IGJ7YOX42OXKPW2SXS7BS4LOX3252LI5TXFBSLGA4UJGYQ, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.429228616 (532h13m15.300311671s) [noise=true , address=145.79.227.37:4565 , piece_id=ACOXUNZEBULHFJKCECKNH53HVJCRALTXWJTDVXMPDYSHEP3WI7ZA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.429956953 (532h13m15.301040018s) [noise=true , address=145.79.227.17:4971 , piece_id=MCU5MAA3TNQ2JQJPI7COBKWDCIMRTXYGHOZVUOFMCM2FC76IK2YQ, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.431262489 (532h13m15.302345574s) [noise=true , address=145.79.227.47:4267 , piece_id=B5RVDJJW7SQSUAFFMOGLILQIWIKVKXZ23ADIZH6OPR2TPCDYHXHA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.432796540 (532h13m15.303879575s) [noise=true , address=145.79.227.30:4761 , piece_id=Y2ECXZUBP74LZU25U54LV57YYWGGVDNQ42BMLT62M7HCLQZGJQWA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.434762945 (532h13m15.30584624s) [noise=true , address=145.79.227.11:4731 , piece_id=V7MENKXI2AAPYVMMIRBCSHCR6ULPMJ7GFFI47OQVBQUOZYTMHU3A, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.439800126 (532h13m15.310883181s) [noise=true , address=145.79.227.51:4410 , piece_id=SAIOGZES5YWRJFKT6MER7CJ4JF4CL2RM2BFSHPD53HWFRTHOOZAQ, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.441923539 (532h13m15.313006584s) [noise=true , address=145.79.227.9:4365 , piece_id=K2NSPPIYWYYGGIU5MXXOYYMEAAWJJOT27RBKK2RDDDKQNRTSK54Q, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.444165699 (532h13m15.315248734s) [noise=true , address=77.90.31.13:4629 , piece_id=TEMDFCEMWM4AVEHFNHXZXMFZ232EMGXPESVDV4HIIHT4ASGKDKJA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:15.446168362 (532h13m15.317251497s) [noise=true , address=145.79.227.10:4403 , piece_id=D2U6KLTGKZFXTWTM4346RBBAISWW6NXBK6SDKWME5REAOQI7XRJA, seg_pos={0 0}, upload=466760]: Uploading piece... Dec 27 19:42:17.039841032 (532h13m16.910924067s) [upload=466760]: Next returned nil segment Dec 27 19:42:17.039858514 (532h13m16.91094155s) [upload=466760]: Waiting for error group managing segments... Dec 27 19:42:17.421723725 (532h13m17.29280676s) [noise=true , address=145.79.227.46:4602 , piece_id=ODRK3VGLSJSKDAMGAEKOGEHAXQOS7TPFJOI2LKVOPARJMPGG435A, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.443821032 (532h13m17.314904077s) [noise=true , address=145.79.227.20:4438 , piece_id=RJDI5FQDC4IMYCATOJPKORLUNHO7NIDU6ZSRZCP5HRLWOOQHHLSA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.444623550 (532h13m17.315706595s) [noise=true , address=145.79.227.10:4403 , piece_id=D2U6KLTGKZFXTWTM4346RBBAISWW6NXBK6SDKWME5REAOQI7XRJA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.446620302 (532h13m17.317703337s) [noise=true , address=145.79.227.37:4565 , piece_id=ACOXUNZEBULHFJKCECKNH53HVJCRALTXWJTDVXMPDYSHEP3WI7ZA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.450253079 (532h13m17.321336124s) [noise=true , address=145.79.227.51:4410 , piece_id=SAIOGZES5YWRJFKT6MER7CJ4JF4CL2RM2BFSHPD53HWFRTHOOZAQ, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.450472247 (532h13m17.321555282s) [noise=true , address=145.79.227.31:4799 , piece_id=FRYDBCIBVM6ABHTMJJORMVZC736K3RIINWX7VPNVN5Q2HCSKR53A, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.450774011 (532h13m17.321857046s) [noise=true , address=145.79.227.47:4267 , piece_id=B5RVDJJW7SQSUAFFMOGLILQIWIKVKXZ23ADIZH6OPR2TPCDYHXHA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.455679783 (532h13m17.326762818s) [noise=true , address=145.79.227.11:4731 , piece_id=V7MENKXI2AAPYVMMIRBCSHCR6ULPMJ7GFFI47OQVBQUOZYTMHU3A, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.459253097 (532h13m17.330336132s) [noise=true , address=145.79.227.22:4510 , piece_id=EW7GAA3RVZDKKPRVXBYYLXOBTHHPSNHVCIUPWVXCCVBDXJP6BN7A, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.462103215 (532h13m17.33318625s) [noise=true , address=145.79.227.45:4231 , piece_id=WNQUP6IGJ7YOX42OXKPW2SXS7BS4LOX3252LI5TXFBSLGA4UJGYQ, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.475778114 (532h13m17.346861149s) [noise=true , address=145.79.227.9:4365 , piece_id=K2NSPPIYWYYGGIU5MXXOYYMEAAWJJOT27RBKK2RDDDKQNRTSK54Q, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.678570046 (532h13m17.549653081s) [noise=true , address=45.91.116.5:4300 , piece_id=GUE4PD4H222CDIGX7EPTDN2HR5EDWAJGP4IHX4VCZKN6MWTW4W2Q, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.692586176 (532h13m17.563669211s) [noise=true , address=77.90.31.13:4629 , piece_id=TEMDFCEMWM4AVEHFNHXZXMFZ232EMGXPESVDV4HIIHT4ASGKDKJA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: <nil> Dec 27 19:42:17.692611213 (532h13m17.563694278s) [seg_pos={0 0}, upload=466760]: Segment reached optimal threshold of 13 pieces. Dec 27 19:42:17.692797358 (532h13m17.563880393s) [noise=true , address=145.79.227.17:4971 , piece_id=MCU5MAA3TNQ2JQJPI7COBKWDCIMRTXYGHOZVUOFMCM2FC76IK2YQ, seg_pos={0 0}, upload=466760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:44422->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.17:4971: use of closed network connection Dec 27 19:42:17.692803690 (532h13m17.563886725s) [noise=true , address=145.79.227.17:4971 , piece_id=MCU5MAA3TNQ2JQJPI7COBKWDCIMRTXYGHOZVUOFMCM2FC76IK2YQ, seg_pos={0 0}, upload=466760]: Long tail cancellation due to optimal threshold Dec 27 19:42:17.693008400 (532h13m17.564091445s) [noise=true , address=145.79.227.30:4761 , piece_id=Y2ECXZUBP74LZU25U54LV57YYWGGVDNQ42BMLT62M7HCLQZGJQWA, seg_pos={0 0}, upload=466760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 19:42:17.693014311 (532h13m17.564097346s) [noise=true , address=145.79.227.30:4761 , piece_id=Y2ECXZUBP74LZU25U54LV57YYWGGVDNQ42BMLT62M7HCLQZGJQWA, seg_pos={0 0}, upload=466760]: Long tail cancellation due to optimal threshold Dec 27 19:42:17.693114512 (532h13m17.564197557s) [noise=true , address=45.91.116.12:4226 , piece_id=FDTTRHXECLDRCGRIBOISE25D34SKPQ4MUGI2AZ3QHXEXCU4MZT5Q, seg_pos={0 0}, upload=466760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 19:42:17.693120313 (532h13m17.564203348s) [noise=true , address=45.91.116.12:4226 , piece_id=FDTTRHXECLDRCGRIBOISE25D34SKPQ4MUGI2AZ3QHXEXCU4MZT5Q, seg_pos={0 0}, upload=466760]: Long tail cancellation due to optimal threshold Dec 27 19:42:17.693138026 (532h13m17.564221061s) [seg_pos={0 0}, upload=466760]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:42:17.693206206 (532h13m17.564289251s) [upload=466760]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:42:17.693211196 (532h13m17.564294241s) [upload=466760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:42:17.982000192 (532h13m17.853083227s) [upload=466760]: Done uploading segments... Dec 27 19:42:17.982019458 (532h13m17.853102493s) [upload=466760]: Upload finished. err: <nil> Duration: 30.002790823s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.485118313s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:42:21 UTC) orbiter warning: context deadline exceeded Duration: 3.485134314s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.655354484s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:41:57 UTC) orbiter warning: context deadline exceeded Duration: 3.655439525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.569589391s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:41:38 UTC) orbiter warning: context deadline exceeded Duration: 3.569609619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.........X....XXX..XX....X...XXXX...X..X..........X..X.] duration: 8.685575957s trace: https://jaeger.dev.storj.io/trace/64d94c49f3d1b149 FAIL (2025-12-27 19:41:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: NCFMBNAXLTK36KNOU7JQ55POCOBYTPROD625GYTE4XVKQY5BOUJA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSUHOF2KIIIEWI3ILXOG7CRNZX7J2VYJOV66YMJAVIGK2K7SDWQ) context deadline exceeded ======LOGS====== Dec 27 19:40:15.421164630 (532h11m15.292247675s) [upload=466734]: Starting upload Dec 27 19:40:15.421224014 (532h11m15.292307059s) [upload=466734]: Done starting upload Dec 27 19:40:15.421344202 (532h11m15.292427237s) [upload=466734]: Uploading segments... Dec 27 19:40:15.421483968 (532h11m15.292567013s) [upload=466734]: Got next segment. Inline: false Dec 27 19:40:15.421551016 (532h11m15.292634071s) [upload=466734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:40:15.421560685 (532h11m15.29264373s) [upload=466734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:40:15.664258334 (532h11m15.535341379s) [seg_pos={0 0}, upload=466734]: Begin upload segment... Dec 27 19:40:15.664319261 (532h11m15.535402296s) [seg_pos={0 0}, upload=466734]: Done begin upload segment. Dec 27 19:40:15.664354568 (532h11m15.535437613s) [noise=true , address=145.79.227.23:4546 , piece_id=WXAP3JG3Y4YXVP6E4MSQBGFA2D5F64LFUR2SJIQAYKNPUGWJNFAQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.664436503 (532h11m15.535519559s) [noise=true , address=145.79.227.42:4122 , piece_id=AYB3XPVLCLTEOTHAJKKFNJFVDW74JRRQ5QX6OIVPKUHLY3HV7F5Q, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.665088515 (532h11m15.53617156s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSUHOF2KIIIEWI3ILXOG7CRNZX7J2VYJOV66YMJAVIGK2K7SDWQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.665565404 (532h11m15.536648449s) [noise=true , address=45.91.116.5:4299 , piece_id=ITNSUWB5NFINIA5TA6BU5Y557JLVFSCOEX4MWPOZKQU6H5R43QIQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.665995282 (532h11m15.537078317s) [noise=true , address=77.90.31.10:4520 , piece_id=YJP3XLZZHI2FAL367IKPACFL6I3ZTTNXUM6FJACX6C25J3UYQCTA, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.666008777 (532h11m15.537091832s) [noise=true , address=145.79.227.37:4567 , piece_id=SBAVSS2ESWY6OQSC4O6M6E2MHJCET3ZGFFBVPDMV7XFTXNBNEODQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.666552713 (532h11m15.537635758s) [noise=true , address=145.79.227.47:4266 , piece_id=3GLPIL7TPSG2RXI3DEWVJHBYU5G5YZ6K7JSYWVB4GNZNGZPEZ6AA, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.666633567 (532h11m15.537716612s) [noise=true , address=145.79.227.48:4302 , piece_id=RA7IYJMTZRXD76C3RU7YEPYBSB5M6ZMTXFSEIPBZNBQSZ5URFKZQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667001598 (532h11m15.538084633s) [noise=true , address=145.79.227.9:4365 , piece_id=4LU46AGGUML6DIRT366EN2HNNW557T2U2E7QWQVKAUOONFXTIYJQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667071892 (532h11m15.538154937s) [noise=true , address=145.79.227.8:4329 , piece_id=AADJD2QNXPUHXQGIJHTW5SGTT4WBJOCS7DBYMD4JOLUZ4ARJSNDQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667354089 (532h11m15.538437134s) [noise=true , address=145.79.227.45:4231 , piece_id=5HJWFZJE6NPUQ6HTGOG3MBJNCBHPY5VHH4INL3VKJZEHYSA54GKA, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667408873 (532h11m15.538491908s) [noise=true , address=145.79.227.26:4653 , piece_id=YD4QIQUPIEXQAZIGX42EMWUCYAOL5LLWHNCC6H6FBJAKZEP6ZY6Q, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667714084 (532h11m15.538797129s) [noise=true , address=45.91.116.6:4975 , piece_id=NCFMBNAXLTK36KNOU7JQ55POCOBYTPROD625GYTE4XVKQY5BOUJA, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.668124346 (532h11m15.539207391s) [noise=true , address=145.79.227.40:4051 , piece_id=66TJFDJ7NRY4AV2TML2ZZMW3QW42V7THHAKHL43ZBAECMRQRHH6Q, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.668464414 (532h11m15.539547449s) [noise=true , address=145.79.227.16:4910 , piece_id=WWLTOXEUMRPUXQQOZXOPMOA6BGNJ7O2SXV2AHOZ7QBI66HDT6H4A, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.669611558 (532h11m15.540694593s) [noise=true , address=145.79.227.36:4517 , piece_id=5Z56OQVBQFGKCNQYJQSKL2U3KVSRVA6RXBHA7NVVWPZZPZX33T4A, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:17.275262726 (532h11m17.146345771s) [upload=466734]: Next returned nil segment Dec 27 19:40:17.275278246 (532h11m17.146361281s) [upload=466734]: Waiting for error group managing segments... Dec 27 19:40:17.660427199 (532h11m17.531510244s) [noise=true , address=145.79.227.23:4546 , piece_id=WXAP3JG3Y4YXVP6E4MSQBGFA2D5F64LFUR2SJIQAYKNPUGWJNFAQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.660495780 (532h11m17.531578825s) [noise=true , address=145.79.227.48:4302 , piece_id=RA7IYJMTZRXD76C3RU7YEPYBSB5M6ZMTXFSEIPBZNBQSZ5URFKZQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.669970832 (532h11m17.541053877s) [noise=true , address=145.79.227.8:4329 , piece_id=AADJD2QNXPUHXQGIJHTW5SGTT4WBJOCS7DBYMD4JOLUZ4ARJSNDQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.673902208 (532h11m17.544985243s) [noise=true , address=145.79.227.42:4122 , piece_id=AYB3XPVLCLTEOTHAJKKFNJFVDW74JRRQ5QX6OIVPKUHLY3HV7F5Q, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.676150249 (532h11m17.547233294s) [noise=true , address=145.79.227.45:4231 , piece_id=5HJWFZJE6NPUQ6HTGOG3MBJNCBHPY5VHH4INL3VKJZEHYSA54GKA, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.685336701 (532h11m17.556419736s) [noise=true , address=145.79.227.47:4266 , piece_id=3GLPIL7TPSG2RXI3DEWVJHBYU5G5YZ6K7JSYWVB4GNZNGZPEZ6AA, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.686438308 (532h11m17.557521343s) [noise=true , address=145.79.227.37:4567 , piece_id=SBAVSS2ESWY6OQSC4O6M6E2MHJCET3ZGFFBVPDMV7XFTXNBNEODQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.686903825 (532h11m17.55798686s) [noise=true , address=145.79.227.36:4517 , piece_id=5Z56OQVBQFGKCNQYJQSKL2U3KVSRVA6RXBHA7NVVWPZZPZX33T4A, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.693236754 (532h11m17.5643198s) [noise=true , address=145.79.227.16:4910 , piece_id=WWLTOXEUMRPUXQQOZXOPMOA6BGNJ7O2SXV2AHOZ7QBI66HDT6H4A, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.693963257 (532h11m17.565046292s) [noise=true , address=145.79.227.9:4365 , piece_id=4LU46AGGUML6DIRT366EN2HNNW557T2U2E7QWQVKAUOONFXTIYJQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.703998667 (532h11m17.575081702s) [noise=true , address=145.79.227.26:4653 , piece_id=YD4QIQUPIEXQAZIGX42EMWUCYAOL5LLWHNCC6H6FBJAKZEP6ZY6Q, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.888126579 (532h11m17.759209624s) [noise=true , address=45.91.116.6:4975 , piece_id=NCFMBNAXLTK36KNOU7JQ55POCOBYTPROD625GYTE4XVKQY5BOUJA, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.921680272 (532h11m17.792763307s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSUHOF2KIIIEWI3ILXOG7CRNZX7J2VYJOV66YMJAVIGK2K7SDWQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.921704038 (532h11m17.792787073s) [seg_pos={0 0}, upload=466734]: Segment reached optimal threshold of 13 pieces. Dec 27 19:40:17.921991885 (532h11m17.79307492s) [noise=true , address=145.79.227.40:4051 , piece_id=66TJFDJ7NRY4AV2TML2ZZMW3QW42V7THHAKHL43ZBAECMRQRHH6Q, seg_pos={0 0}, upload=466734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 19:40:17.922000782 (532h11m17.793083817s) [noise=true , address=145.79.227.40:4051 , piece_id=66TJFDJ7NRY4AV2TML2ZZMW3QW42V7THHAKHL43ZBAECMRQRHH6Q, seg_pos={0 0}, upload=466734]: Long tail cancellation due to optimal threshold Dec 27 19:40:17.922367551 (532h11m17.793450596s) [noise=true , address=77.90.31.10:4520 , piece_id=YJP3XLZZHI2FAL367IKPACFL6I3ZTTNXUM6FJACX6C25J3UYQCTA, seg_pos={0 0}, upload=466734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 19:40:17.922406104 (532h11m17.793489149s) [noise=true , address=77.90.31.10:4520 , piece_id=YJP3XLZZHI2FAL367IKPACFL6I3ZTTNXUM6FJACX6C25J3UYQCTA, seg_pos={0 0}, upload=466734]: Long tail cancellation due to optimal threshold Dec 27 19:40:17.922415281 (532h11m17.793498326s) [noise=true , address=45.91.116.5:4299 , piece_id=ITNSUWB5NFINIA5TA6BU5Y557JLVFSCOEX4MWPOZKQU6H5R43QIQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 19:40:17.922424258 (532h11m17.793507293s) [noise=true , address=45.91.116.5:4299 , piece_id=ITNSUWB5NFINIA5TA6BU5Y557JLVFSCOEX4MWPOZKQU6H5R43QIQ, seg_pos={0 0}, upload=466734]: Long tail cancellation due to optimal threshold Dec 27 19:40:17.922443345 (532h11m17.79352639s) [seg_pos={0 0}, upload=466734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:40:17.922546131 (532h11m17.793629166s) [upload=466734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:40:17.922552142 (532h11m17.793635177s) [upload=466734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:40:18.213084127 (532h11m18.084167162s) [upload=466734]: Done uploading segments... Dec 27 19:40:18.213102202 (532h11m18.084185237s) [upload=466734]: Upload finished. err: <nil> Duration: 8.68568204s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.58042288s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:41:13 UTC) orbiter warning: context deadline exceeded Duration: 3.580471693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X.........X....XXX..XX....X...XXXX...X..X..........X..X] duration: 30.003201644s trace: https://jaeger.dev.storj.io/trace/64d94c49f3d1b149 FAIL (2025-12-27 19:40:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: NCFMBNAXLTK36KNOU7JQ55POCOBYTPROD625GYTE4XVKQY5BOUJA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSUHOF2KIIIEWI3ILXOG7CRNZX7J2VYJOV66YMJAVIGK2K7SDWQ) context deadline exceeded ======LOGS====== Dec 27 19:40:15.421164630 (532h11m15.292247675s) [upload=466734]: Starting upload Dec 27 19:40:15.421224014 (532h11m15.292307059s) [upload=466734]: Done starting upload Dec 27 19:40:15.421344202 (532h11m15.292427237s) [upload=466734]: Uploading segments... Dec 27 19:40:15.421483968 (532h11m15.292567013s) [upload=466734]: Got next segment. Inline: false Dec 27 19:40:15.421551016 (532h11m15.292634071s) [upload=466734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:40:15.421560685 (532h11m15.29264373s) [upload=466734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:40:15.664258334 (532h11m15.535341379s) [seg_pos={0 0}, upload=466734]: Begin upload segment... Dec 27 19:40:15.664319261 (532h11m15.535402296s) [seg_pos={0 0}, upload=466734]: Done begin upload segment. Dec 27 19:40:15.664354568 (532h11m15.535437613s) [noise=true , address=145.79.227.23:4546 , piece_id=WXAP3JG3Y4YXVP6E4MSQBGFA2D5F64LFUR2SJIQAYKNPUGWJNFAQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.664436503 (532h11m15.535519559s) [noise=true , address=145.79.227.42:4122 , piece_id=AYB3XPVLCLTEOTHAJKKFNJFVDW74JRRQ5QX6OIVPKUHLY3HV7F5Q, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.665088515 (532h11m15.53617156s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSUHOF2KIIIEWI3ILXOG7CRNZX7J2VYJOV66YMJAVIGK2K7SDWQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.665565404 (532h11m15.536648449s) [noise=true , address=45.91.116.5:4299 , piece_id=ITNSUWB5NFINIA5TA6BU5Y557JLVFSCOEX4MWPOZKQU6H5R43QIQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.665995282 (532h11m15.537078317s) [noise=true , address=77.90.31.10:4520 , piece_id=YJP3XLZZHI2FAL367IKPACFL6I3ZTTNXUM6FJACX6C25J3UYQCTA, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.666008777 (532h11m15.537091832s) [noise=true , address=145.79.227.37:4567 , piece_id=SBAVSS2ESWY6OQSC4O6M6E2MHJCET3ZGFFBVPDMV7XFTXNBNEODQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.666552713 (532h11m15.537635758s) [noise=true , address=145.79.227.47:4266 , piece_id=3GLPIL7TPSG2RXI3DEWVJHBYU5G5YZ6K7JSYWVB4GNZNGZPEZ6AA, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.666633567 (532h11m15.537716612s) [noise=true , address=145.79.227.48:4302 , piece_id=RA7IYJMTZRXD76C3RU7YEPYBSB5M6ZMTXFSEIPBZNBQSZ5URFKZQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667001598 (532h11m15.538084633s) [noise=true , address=145.79.227.9:4365 , piece_id=4LU46AGGUML6DIRT366EN2HNNW557T2U2E7QWQVKAUOONFXTIYJQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667071892 (532h11m15.538154937s) [noise=true , address=145.79.227.8:4329 , piece_id=AADJD2QNXPUHXQGIJHTW5SGTT4WBJOCS7DBYMD4JOLUZ4ARJSNDQ, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667354089 (532h11m15.538437134s) [noise=true , address=145.79.227.45:4231 , piece_id=5HJWFZJE6NPUQ6HTGOG3MBJNCBHPY5VHH4INL3VKJZEHYSA54GKA, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667408873 (532h11m15.538491908s) [noise=true , address=145.79.227.26:4653 , piece_id=YD4QIQUPIEXQAZIGX42EMWUCYAOL5LLWHNCC6H6FBJAKZEP6ZY6Q, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.667714084 (532h11m15.538797129s) [noise=true , address=45.91.116.6:4975 , piece_id=NCFMBNAXLTK36KNOU7JQ55POCOBYTPROD625GYTE4XVKQY5BOUJA, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.668124346 (532h11m15.539207391s) [noise=true , address=145.79.227.40:4051 , piece_id=66TJFDJ7NRY4AV2TML2ZZMW3QW42V7THHAKHL43ZBAECMRQRHH6Q, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.668464414 (532h11m15.539547449s) [noise=true , address=145.79.227.16:4910 , piece_id=WWLTOXEUMRPUXQQOZXOPMOA6BGNJ7O2SXV2AHOZ7QBI66HDT6H4A, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:15.669611558 (532h11m15.540694593s) [noise=true , address=145.79.227.36:4517 , piece_id=5Z56OQVBQFGKCNQYJQSKL2U3KVSRVA6RXBHA7NVVWPZZPZX33T4A, seg_pos={0 0}, upload=466734]: Uploading piece... Dec 27 19:40:17.275262726 (532h11m17.146345771s) [upload=466734]: Next returned nil segment Dec 27 19:40:17.275278246 (532h11m17.146361281s) [upload=466734]: Waiting for error group managing segments... Dec 27 19:40:17.660427199 (532h11m17.531510244s) [noise=true , address=145.79.227.23:4546 , piece_id=WXAP3JG3Y4YXVP6E4MSQBGFA2D5F64LFUR2SJIQAYKNPUGWJNFAQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.660495780 (532h11m17.531578825s) [noise=true , address=145.79.227.48:4302 , piece_id=RA7IYJMTZRXD76C3RU7YEPYBSB5M6ZMTXFSEIPBZNBQSZ5URFKZQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.669970832 (532h11m17.541053877s) [noise=true , address=145.79.227.8:4329 , piece_id=AADJD2QNXPUHXQGIJHTW5SGTT4WBJOCS7DBYMD4JOLUZ4ARJSNDQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.673902208 (532h11m17.544985243s) [noise=true , address=145.79.227.42:4122 , piece_id=AYB3XPVLCLTEOTHAJKKFNJFVDW74JRRQ5QX6OIVPKUHLY3HV7F5Q, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.676150249 (532h11m17.547233294s) [noise=true , address=145.79.227.45:4231 , piece_id=5HJWFZJE6NPUQ6HTGOG3MBJNCBHPY5VHH4INL3VKJZEHYSA54GKA, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.685336701 (532h11m17.556419736s) [noise=true , address=145.79.227.47:4266 , piece_id=3GLPIL7TPSG2RXI3DEWVJHBYU5G5YZ6K7JSYWVB4GNZNGZPEZ6AA, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.686438308 (532h11m17.557521343s) [noise=true , address=145.79.227.37:4567 , piece_id=SBAVSS2ESWY6OQSC4O6M6E2MHJCET3ZGFFBVPDMV7XFTXNBNEODQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.686903825 (532h11m17.55798686s) [noise=true , address=145.79.227.36:4517 , piece_id=5Z56OQVBQFGKCNQYJQSKL2U3KVSRVA6RXBHA7NVVWPZZPZX33T4A, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.693236754 (532h11m17.5643198s) [noise=true , address=145.79.227.16:4910 , piece_id=WWLTOXEUMRPUXQQOZXOPMOA6BGNJ7O2SXV2AHOZ7QBI66HDT6H4A, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.693963257 (532h11m17.565046292s) [noise=true , address=145.79.227.9:4365 , piece_id=4LU46AGGUML6DIRT366EN2HNNW557T2U2E7QWQVKAUOONFXTIYJQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.703998667 (532h11m17.575081702s) [noise=true , address=145.79.227.26:4653 , piece_id=YD4QIQUPIEXQAZIGX42EMWUCYAOL5LLWHNCC6H6FBJAKZEP6ZY6Q, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.888126579 (532h11m17.759209624s) [noise=true , address=45.91.116.6:4975 , piece_id=NCFMBNAXLTK36KNOU7JQ55POCOBYTPROD625GYTE4XVKQY5BOUJA, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.921680272 (532h11m17.792763307s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSUHOF2KIIIEWI3ILXOG7CRNZX7J2VYJOV66YMJAVIGK2K7SDWQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: <nil> Dec 27 19:40:17.921704038 (532h11m17.792787073s) [seg_pos={0 0}, upload=466734]: Segment reached optimal threshold of 13 pieces. Dec 27 19:40:17.921991885 (532h11m17.79307492s) [noise=true , address=145.79.227.40:4051 , piece_id=66TJFDJ7NRY4AV2TML2ZZMW3QW42V7THHAKHL43ZBAECMRQRHH6Q, seg_pos={0 0}, upload=466734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 19:40:17.922000782 (532h11m17.793083817s) [noise=true , address=145.79.227.40:4051 , piece_id=66TJFDJ7NRY4AV2TML2ZZMW3QW42V7THHAKHL43ZBAECMRQRHH6Q, seg_pos={0 0}, upload=466734]: Long tail cancellation due to optimal threshold Dec 27 19:40:17.922367551 (532h11m17.793450596s) [noise=true , address=77.90.31.10:4520 , piece_id=YJP3XLZZHI2FAL367IKPACFL6I3ZTTNXUM6FJACX6C25J3UYQCTA, seg_pos={0 0}, upload=466734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 19:40:17.922406104 (532h11m17.793489149s) [noise=true , address=77.90.31.10:4520 , piece_id=YJP3XLZZHI2FAL367IKPACFL6I3ZTTNXUM6FJACX6C25J3UYQCTA, seg_pos={0 0}, upload=466734]: Long tail cancellation due to optimal threshold Dec 27 19:40:17.922415281 (532h11m17.793498326s) [noise=true , address=45.91.116.5:4299 , piece_id=ITNSUWB5NFINIA5TA6BU5Y557JLVFSCOEX4MWPOZKQU6H5R43QIQ, seg_pos={0 0}, upload=466734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 19:40:17.922424258 (532h11m17.793507293s) [noise=true , address=45.91.116.5:4299 , piece_id=ITNSUWB5NFINIA5TA6BU5Y557JLVFSCOEX4MWPOZKQU6H5R43QIQ, seg_pos={0 0}, upload=466734]: Long tail cancellation due to optimal threshold Dec 27 19:40:17.922443345 (532h11m17.79352639s) [seg_pos={0 0}, upload=466734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:40:17.922546131 (532h11m17.793629166s) [upload=466734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:40:17.922552142 (532h11m17.793635177s) [upload=466734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:40:18.213084127 (532h11m18.084167162s) [upload=466734]: Done uploading segments... Dec 27 19:40:18.213102202 (532h11m18.084185237s) [upload=466734]: Upload finished. err: <nil> Duration: 30.003224749s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.571420466s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:40:09 UTC) orbiter warning: context deadline exceeded Duration: 3.571465311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.546003751s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:39:24 UTC) orbiter warning: context deadline exceeded Duration: 3.546023117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.....X.........X....XXX..XX....X...XXXX...X..X..........X..] duration: 5.685913733s trace: https://jaeger.dev.storj.io/trace/30bb311b90add84d FAIL (2025-12-27 19:39:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:37:48.690630061 (532h8m48.561713096s) [upload=466697]: Starting upload Dec 27 19:37:48.690712669 (532h8m48.561795704s) [upload=466697]: Done starting upload Dec 27 19:37:48.690955391 (532h8m48.562038426s) [upload=466697]: Uploading segments... Dec 27 19:37:48.691029301 (532h8m48.562112336s) [upload=466697]: Got next segment. Inline: false Dec 27 19:37:48.691068776 (532h8m48.562151821s) [upload=466697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:37:48.691073515 (532h8m48.56215656s) [upload=466697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:37:48.935556968 (532h8m48.806640013s) [seg_pos={0 0}, upload=466697]: Begin upload segment... Dec 27 19:37:48.935613636 (532h8m48.806696671s) [seg_pos={0 0}, upload=466697]: Done begin upload segment. Dec 27 19:37:48.935647290 (532h8m48.806730335s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.935736880 (532h8m48.806819915s) [noise=true , address=145.79.227.38:4979 , piece_id=7HDGUQMD6XZRKTGQW7TIPO6CSATM7D6ZMSUP4EL3K437JLYDGFFQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.936280285 (532h8m48.80736332s) [noise=true , address=145.79.227.19:4078 , piece_id=RKQLKVPP36PGPNJ4DKOPP5WDRCOB6A7OBPKJLI6FOL3LIBYRN6GA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.936735853 (532h8m48.807818888s) [noise=true , address=145.79.227.25:4619 , piece_id=J4FUKC5TU3UNZFCEIJIX3ZHIIUFLKRQEAKIQQGKYKXGASF35URRA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.937239000 (532h8m48.808322045s) [noise=true , address=145.79.227.39:4013 , piece_id=KRDY4PTT5AOWXDX3XWT6ZXAQAZKJPJFAPIEFA6Y2OYORRS3IAVQA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.937757217 (532h8m48.808840262s) [noise=true , address=145.79.227.2:4114 , piece_id=5Q63YGX3H4H7G46OSQXRBGAUW575FL5XQIKI3RMRQEZH44MTRLJQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.938392146 (532h8m48.809475191s) [noise=true , address=145.79.227.18:4007 , piece_id=6ZNRQBPOASDGUJ5L7YBLWEG3XNQDKFSPMUFBNPLBNSSU2JYXGJFQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.940268218 (532h8m48.811351254s) [noise=true , address=77.90.31.19:4842 , piece_id=43EZ4Y3RFQA73HY67VBNIW25XS47VKIZ5XLH7Z54ILTBY76LXHBQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.940549103 (532h8m48.811632138s) [noise=true , address=145.79.227.4:4186 , piece_id=TKFMDABYGQER7QPXSRIKCZCZPZIGCRIJOLQLY6BWUFHVJWX234CA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.941541924 (532h8m48.812624959s) [noise=true , address=145.79.227.13:4804 , piece_id=VD4CNQ4QX46SZV5ITIURAWOUNCRFD55YEC2EQLW263YYKDHKWXNQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.942235926 (532h8m48.813318971s) [noise=true , address=145.79.227.14:4839 , piece_id=7HOUIUK7CKPW4APUHECVADPTLI275E5SA5KFD7SIDJGBWK6EELSQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.943152731 (532h8m48.814235776s) [noise=true , address=45.91.116.11:4119 , piece_id=B4PJ5KTD47LF7BJP3OM2QKALW4VHTKVA47D4OGMEWS6YI4VTMNLQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.944338589 (532h8m48.815421624s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.944736858 (532h8m48.815819893s) [noise=true , address=145.79.227.46:4603 , piece_id=DK3CXDD6Z7ZAWKKWHAQGFG7QD3ZWDWUTI4KB6EMKS6ELD5YM7I6Q, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.946668406 (532h8m48.817751451s) [noise=true , address=145.79.227.36:4519 , piece_id=33JES4TEDY7F75QSKUIQX2P2B7CNMW4JAJYAFEUZBAKOYFE5VQQA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.950681046 (532h8m48.821764101s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:50.647802234 (532h8m50.518885279s) [upload=466697]: Next returned nil segment Dec 27 19:37:50.647817072 (532h8m50.518900117s) [upload=466697]: Waiting for error group managing segments... Dec 27 19:37:50.933368024 (532h8m50.804451069s) [noise=true , address=145.79.227.2:4114 , piece_id=5Q63YGX3H4H7G46OSQXRBGAUW575FL5XQIKI3RMRQEZH44MTRLJQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.941620428 (532h8m50.812703473s) [noise=true , address=145.79.227.25:4619 , piece_id=J4FUKC5TU3UNZFCEIJIX3ZHIIUFLKRQEAKIQQGKYKXGASF35URRA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.942298189 (532h8m50.813381224s) [noise=true , address=145.79.227.19:4078 , piece_id=RKQLKVPP36PGPNJ4DKOPP5WDRCOB6A7OBPKJLI6FOL3LIBYRN6GA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.948633703 (532h8m50.819716738s) [noise=true , address=145.79.227.38:4979 , piece_id=7HDGUQMD6XZRKTGQW7TIPO6CSATM7D6ZMSUP4EL3K437JLYDGFFQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.950374718 (532h8m50.821457763s) [noise=true , address=145.79.227.4:4186 , piece_id=TKFMDABYGQER7QPXSRIKCZCZPZIGCRIJOLQLY6BWUFHVJWX234CA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.959651914 (532h8m50.830734949s) [noise=true , address=145.79.227.18:4007 , piece_id=6ZNRQBPOASDGUJ5L7YBLWEG3XNQDKFSPMUFBNPLBNSSU2JYXGJFQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.960881976 (532h8m50.831965011s) [noise=true , address=145.79.227.39:4013 , piece_id=KRDY4PTT5AOWXDX3XWT6ZXAQAZKJPJFAPIEFA6Y2OYORRS3IAVQA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.966459076 (532h8m50.837542111s) [noise=true , address=145.79.227.13:4804 , piece_id=VD4CNQ4QX46SZV5ITIURAWOUNCRFD55YEC2EQLW263YYKDHKWXNQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.975737935 (532h8m50.84682097s) [noise=true , address=145.79.227.46:4603 , piece_id=DK3CXDD6Z7ZAWKKWHAQGFG7QD3ZWDWUTI4KB6EMKS6ELD5YM7I6Q, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.979030885 (532h8m50.85011393s) [noise=true , address=145.79.227.36:4519 , piece_id=33JES4TEDY7F75QSKUIQX2P2B7CNMW4JAJYAFEUZBAKOYFE5VQQA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.114036691 (532h8m50.985119727s) [noise=true , address=145.79.227.14:4839 , piece_id=7HOUIUK7CKPW4APUHECVADPTLI275E5SA5KFD7SIDJGBWK6EELSQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.147172218 (532h8m51.018255263s) [noise=true , address=77.90.31.19:4842 , piece_id=43EZ4Y3RFQA73HY67VBNIW25XS47VKIZ5XLH7Z54ILTBY76LXHBQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.182458229 (532h8m51.053541274s) [noise=true , address=45.91.116.11:4119 , piece_id=B4PJ5KTD47LF7BJP3OM2QKALW4VHTKVA47D4OGMEWS6YI4VTMNLQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.182485211 (532h8m51.053568256s) [seg_pos={0 0}, upload=466697]: Segment reached optimal threshold of 13 pieces. Dec 27 19:37:51.182765154 (532h8m51.053848199s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.182780683 (532h8m51.053863728s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.182796313 (532h8m51.053879359s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.182807064 (532h8m51.053890109s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.183015981 (532h8m51.054099026s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.183025179 (532h8m51.054108224s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.183045598 (532h8m51.054128643s) [seg_pos={0 0}, upload=466697]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:37:51.183132633 (532h8m51.054215678s) [upload=466697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:37:51.183139386 (532h8m51.054222431s) [upload=466697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:37:51.477064154 (532h8m51.348147209s) [upload=466697]: Done uploading segments... Dec 27 19:37:51.477092258 (532h8m51.348175293s) [upload=466697]: Upload finished. err: <nil> Duration: 5.685940784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.....X.........X....XXX..XX....X...XXXX...X..X..........X.] duration: 5.951155769s trace: https://jaeger.dev.storj.io/trace/30bb311b90add84d FAIL (2025-12-27 19:39:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:37:48.690630061 (532h8m48.561713096s) [upload=466697]: Starting upload Dec 27 19:37:48.690712669 (532h8m48.561795704s) [upload=466697]: Done starting upload Dec 27 19:37:48.690955391 (532h8m48.562038426s) [upload=466697]: Uploading segments... Dec 27 19:37:48.691029301 (532h8m48.562112336s) [upload=466697]: Got next segment. Inline: false Dec 27 19:37:48.691068776 (532h8m48.562151821s) [upload=466697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:37:48.691073515 (532h8m48.56215656s) [upload=466697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:37:48.935556968 (532h8m48.806640013s) [seg_pos={0 0}, upload=466697]: Begin upload segment... Dec 27 19:37:48.935613636 (532h8m48.806696671s) [seg_pos={0 0}, upload=466697]: Done begin upload segment. Dec 27 19:37:48.935647290 (532h8m48.806730335s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.935736880 (532h8m48.806819915s) [noise=true , address=145.79.227.38:4979 , piece_id=7HDGUQMD6XZRKTGQW7TIPO6CSATM7D6ZMSUP4EL3K437JLYDGFFQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.936280285 (532h8m48.80736332s) [noise=true , address=145.79.227.19:4078 , piece_id=RKQLKVPP36PGPNJ4DKOPP5WDRCOB6A7OBPKJLI6FOL3LIBYRN6GA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.936735853 (532h8m48.807818888s) [noise=true , address=145.79.227.25:4619 , piece_id=J4FUKC5TU3UNZFCEIJIX3ZHIIUFLKRQEAKIQQGKYKXGASF35URRA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.937239000 (532h8m48.808322045s) [noise=true , address=145.79.227.39:4013 , piece_id=KRDY4PTT5AOWXDX3XWT6ZXAQAZKJPJFAPIEFA6Y2OYORRS3IAVQA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.937757217 (532h8m48.808840262s) [noise=true , address=145.79.227.2:4114 , piece_id=5Q63YGX3H4H7G46OSQXRBGAUW575FL5XQIKI3RMRQEZH44MTRLJQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.938392146 (532h8m48.809475191s) [noise=true , address=145.79.227.18:4007 , piece_id=6ZNRQBPOASDGUJ5L7YBLWEG3XNQDKFSPMUFBNPLBNSSU2JYXGJFQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.940268218 (532h8m48.811351254s) [noise=true , address=77.90.31.19:4842 , piece_id=43EZ4Y3RFQA73HY67VBNIW25XS47VKIZ5XLH7Z54ILTBY76LXHBQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.940549103 (532h8m48.811632138s) [noise=true , address=145.79.227.4:4186 , piece_id=TKFMDABYGQER7QPXSRIKCZCZPZIGCRIJOLQLY6BWUFHVJWX234CA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.941541924 (532h8m48.812624959s) [noise=true , address=145.79.227.13:4804 , piece_id=VD4CNQ4QX46SZV5ITIURAWOUNCRFD55YEC2EQLW263YYKDHKWXNQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.942235926 (532h8m48.813318971s) [noise=true , address=145.79.227.14:4839 , piece_id=7HOUIUK7CKPW4APUHECVADPTLI275E5SA5KFD7SIDJGBWK6EELSQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.943152731 (532h8m48.814235776s) [noise=true , address=45.91.116.11:4119 , piece_id=B4PJ5KTD47LF7BJP3OM2QKALW4VHTKVA47D4OGMEWS6YI4VTMNLQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.944338589 (532h8m48.815421624s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.944736858 (532h8m48.815819893s) [noise=true , address=145.79.227.46:4603 , piece_id=DK3CXDD6Z7ZAWKKWHAQGFG7QD3ZWDWUTI4KB6EMKS6ELD5YM7I6Q, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.946668406 (532h8m48.817751451s) [noise=true , address=145.79.227.36:4519 , piece_id=33JES4TEDY7F75QSKUIQX2P2B7CNMW4JAJYAFEUZBAKOYFE5VQQA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.950681046 (532h8m48.821764101s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:50.647802234 (532h8m50.518885279s) [upload=466697]: Next returned nil segment Dec 27 19:37:50.647817072 (532h8m50.518900117s) [upload=466697]: Waiting for error group managing segments... Dec 27 19:37:50.933368024 (532h8m50.804451069s) [noise=true , address=145.79.227.2:4114 , piece_id=5Q63YGX3H4H7G46OSQXRBGAUW575FL5XQIKI3RMRQEZH44MTRLJQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.941620428 (532h8m50.812703473s) [noise=true , address=145.79.227.25:4619 , piece_id=J4FUKC5TU3UNZFCEIJIX3ZHIIUFLKRQEAKIQQGKYKXGASF35URRA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.942298189 (532h8m50.813381224s) [noise=true , address=145.79.227.19:4078 , piece_id=RKQLKVPP36PGPNJ4DKOPP5WDRCOB6A7OBPKJLI6FOL3LIBYRN6GA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.948633703 (532h8m50.819716738s) [noise=true , address=145.79.227.38:4979 , piece_id=7HDGUQMD6XZRKTGQW7TIPO6CSATM7D6ZMSUP4EL3K437JLYDGFFQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.950374718 (532h8m50.821457763s) [noise=true , address=145.79.227.4:4186 , piece_id=TKFMDABYGQER7QPXSRIKCZCZPZIGCRIJOLQLY6BWUFHVJWX234CA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.959651914 (532h8m50.830734949s) [noise=true , address=145.79.227.18:4007 , piece_id=6ZNRQBPOASDGUJ5L7YBLWEG3XNQDKFSPMUFBNPLBNSSU2JYXGJFQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.960881976 (532h8m50.831965011s) [noise=true , address=145.79.227.39:4013 , piece_id=KRDY4PTT5AOWXDX3XWT6ZXAQAZKJPJFAPIEFA6Y2OYORRS3IAVQA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.966459076 (532h8m50.837542111s) [noise=true , address=145.79.227.13:4804 , piece_id=VD4CNQ4QX46SZV5ITIURAWOUNCRFD55YEC2EQLW263YYKDHKWXNQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.975737935 (532h8m50.84682097s) [noise=true , address=145.79.227.46:4603 , piece_id=DK3CXDD6Z7ZAWKKWHAQGFG7QD3ZWDWUTI4KB6EMKS6ELD5YM7I6Q, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.979030885 (532h8m50.85011393s) [noise=true , address=145.79.227.36:4519 , piece_id=33JES4TEDY7F75QSKUIQX2P2B7CNMW4JAJYAFEUZBAKOYFE5VQQA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.114036691 (532h8m50.985119727s) [noise=true , address=145.79.227.14:4839 , piece_id=7HOUIUK7CKPW4APUHECVADPTLI275E5SA5KFD7SIDJGBWK6EELSQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.147172218 (532h8m51.018255263s) [noise=true , address=77.90.31.19:4842 , piece_id=43EZ4Y3RFQA73HY67VBNIW25XS47VKIZ5XLH7Z54ILTBY76LXHBQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.182458229 (532h8m51.053541274s) [noise=true , address=45.91.116.11:4119 , piece_id=B4PJ5KTD47LF7BJP3OM2QKALW4VHTKVA47D4OGMEWS6YI4VTMNLQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.182485211 (532h8m51.053568256s) [seg_pos={0 0}, upload=466697]: Segment reached optimal threshold of 13 pieces. Dec 27 19:37:51.182765154 (532h8m51.053848199s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.182780683 (532h8m51.053863728s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.182796313 (532h8m51.053879359s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.182807064 (532h8m51.053890109s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.183015981 (532h8m51.054099026s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.183025179 (532h8m51.054108224s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.183045598 (532h8m51.054128643s) [seg_pos={0 0}, upload=466697]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:37:51.183132633 (532h8m51.054215678s) [upload=466697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:37:51.183139386 (532h8m51.054222431s) [upload=466697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:37:51.477064154 (532h8m51.348147209s) [upload=466697]: Done uploading segments... Dec 27 19:37:51.477092258 (532h8m51.348175293s) [upload=466697]: Upload finished. err: <nil> Duration: 5.95118809s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.422814324s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:38:19 UTC) orbiter warning: context deadline exceeded Duration: 3.42283832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.....X.........X....XXX..XX....X...XXXX...X..X..........X] duration: 30.002910014s trace: https://jaeger.dev.storj.io/trace/30bb311b90add84d FAIL (2025-12-27 19:38:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:37:48.690630061 (532h8m48.561713096s) [upload=466697]: Starting upload Dec 27 19:37:48.690712669 (532h8m48.561795704s) [upload=466697]: Done starting upload Dec 27 19:37:48.690955391 (532h8m48.562038426s) [upload=466697]: Uploading segments... Dec 27 19:37:48.691029301 (532h8m48.562112336s) [upload=466697]: Got next segment. Inline: false Dec 27 19:37:48.691068776 (532h8m48.562151821s) [upload=466697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:37:48.691073515 (532h8m48.56215656s) [upload=466697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:37:48.935556968 (532h8m48.806640013s) [seg_pos={0 0}, upload=466697]: Begin upload segment... Dec 27 19:37:48.935613636 (532h8m48.806696671s) [seg_pos={0 0}, upload=466697]: Done begin upload segment. Dec 27 19:37:48.935647290 (532h8m48.806730335s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.935736880 (532h8m48.806819915s) [noise=true , address=145.79.227.38:4979 , piece_id=7HDGUQMD6XZRKTGQW7TIPO6CSATM7D6ZMSUP4EL3K437JLYDGFFQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.936280285 (532h8m48.80736332s) [noise=true , address=145.79.227.19:4078 , piece_id=RKQLKVPP36PGPNJ4DKOPP5WDRCOB6A7OBPKJLI6FOL3LIBYRN6GA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.936735853 (532h8m48.807818888s) [noise=true , address=145.79.227.25:4619 , piece_id=J4FUKC5TU3UNZFCEIJIX3ZHIIUFLKRQEAKIQQGKYKXGASF35URRA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.937239000 (532h8m48.808322045s) [noise=true , address=145.79.227.39:4013 , piece_id=KRDY4PTT5AOWXDX3XWT6ZXAQAZKJPJFAPIEFA6Y2OYORRS3IAVQA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.937757217 (532h8m48.808840262s) [noise=true , address=145.79.227.2:4114 , piece_id=5Q63YGX3H4H7G46OSQXRBGAUW575FL5XQIKI3RMRQEZH44MTRLJQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.938392146 (532h8m48.809475191s) [noise=true , address=145.79.227.18:4007 , piece_id=6ZNRQBPOASDGUJ5L7YBLWEG3XNQDKFSPMUFBNPLBNSSU2JYXGJFQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.940268218 (532h8m48.811351254s) [noise=true , address=77.90.31.19:4842 , piece_id=43EZ4Y3RFQA73HY67VBNIW25XS47VKIZ5XLH7Z54ILTBY76LXHBQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.940549103 (532h8m48.811632138s) [noise=true , address=145.79.227.4:4186 , piece_id=TKFMDABYGQER7QPXSRIKCZCZPZIGCRIJOLQLY6BWUFHVJWX234CA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.941541924 (532h8m48.812624959s) [noise=true , address=145.79.227.13:4804 , piece_id=VD4CNQ4QX46SZV5ITIURAWOUNCRFD55YEC2EQLW263YYKDHKWXNQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.942235926 (532h8m48.813318971s) [noise=true , address=145.79.227.14:4839 , piece_id=7HOUIUK7CKPW4APUHECVADPTLI275E5SA5KFD7SIDJGBWK6EELSQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.943152731 (532h8m48.814235776s) [noise=true , address=45.91.116.11:4119 , piece_id=B4PJ5KTD47LF7BJP3OM2QKALW4VHTKVA47D4OGMEWS6YI4VTMNLQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.944338589 (532h8m48.815421624s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.944736858 (532h8m48.815819893s) [noise=true , address=145.79.227.46:4603 , piece_id=DK3CXDD6Z7ZAWKKWHAQGFG7QD3ZWDWUTI4KB6EMKS6ELD5YM7I6Q, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.946668406 (532h8m48.817751451s) [noise=true , address=145.79.227.36:4519 , piece_id=33JES4TEDY7F75QSKUIQX2P2B7CNMW4JAJYAFEUZBAKOYFE5VQQA, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:48.950681046 (532h8m48.821764101s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Uploading piece... Dec 27 19:37:50.647802234 (532h8m50.518885279s) [upload=466697]: Next returned nil segment Dec 27 19:37:50.647817072 (532h8m50.518900117s) [upload=466697]: Waiting for error group managing segments... Dec 27 19:37:50.933368024 (532h8m50.804451069s) [noise=true , address=145.79.227.2:4114 , piece_id=5Q63YGX3H4H7G46OSQXRBGAUW575FL5XQIKI3RMRQEZH44MTRLJQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.941620428 (532h8m50.812703473s) [noise=true , address=145.79.227.25:4619 , piece_id=J4FUKC5TU3UNZFCEIJIX3ZHIIUFLKRQEAKIQQGKYKXGASF35URRA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.942298189 (532h8m50.813381224s) [noise=true , address=145.79.227.19:4078 , piece_id=RKQLKVPP36PGPNJ4DKOPP5WDRCOB6A7OBPKJLI6FOL3LIBYRN6GA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.948633703 (532h8m50.819716738s) [noise=true , address=145.79.227.38:4979 , piece_id=7HDGUQMD6XZRKTGQW7TIPO6CSATM7D6ZMSUP4EL3K437JLYDGFFQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.950374718 (532h8m50.821457763s) [noise=true , address=145.79.227.4:4186 , piece_id=TKFMDABYGQER7QPXSRIKCZCZPZIGCRIJOLQLY6BWUFHVJWX234CA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.959651914 (532h8m50.830734949s) [noise=true , address=145.79.227.18:4007 , piece_id=6ZNRQBPOASDGUJ5L7YBLWEG3XNQDKFSPMUFBNPLBNSSU2JYXGJFQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.960881976 (532h8m50.831965011s) [noise=true , address=145.79.227.39:4013 , piece_id=KRDY4PTT5AOWXDX3XWT6ZXAQAZKJPJFAPIEFA6Y2OYORRS3IAVQA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.966459076 (532h8m50.837542111s) [noise=true , address=145.79.227.13:4804 , piece_id=VD4CNQ4QX46SZV5ITIURAWOUNCRFD55YEC2EQLW263YYKDHKWXNQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.975737935 (532h8m50.84682097s) [noise=true , address=145.79.227.46:4603 , piece_id=DK3CXDD6Z7ZAWKKWHAQGFG7QD3ZWDWUTI4KB6EMKS6ELD5YM7I6Q, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:50.979030885 (532h8m50.85011393s) [noise=true , address=145.79.227.36:4519 , piece_id=33JES4TEDY7F75QSKUIQX2P2B7CNMW4JAJYAFEUZBAKOYFE5VQQA, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.114036691 (532h8m50.985119727s) [noise=true , address=145.79.227.14:4839 , piece_id=7HOUIUK7CKPW4APUHECVADPTLI275E5SA5KFD7SIDJGBWK6EELSQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.147172218 (532h8m51.018255263s) [noise=true , address=77.90.31.19:4842 , piece_id=43EZ4Y3RFQA73HY67VBNIW25XS47VKIZ5XLH7Z54ILTBY76LXHBQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.182458229 (532h8m51.053541274s) [noise=true , address=45.91.116.11:4119 , piece_id=B4PJ5KTD47LF7BJP3OM2QKALW4VHTKVA47D4OGMEWS6YI4VTMNLQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: <nil> Dec 27 19:37:51.182485211 (532h8m51.053568256s) [seg_pos={0 0}, upload=466697]: Segment reached optimal threshold of 13 pieces. Dec 27 19:37:51.182765154 (532h8m51.053848199s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.182780683 (532h8m51.053863728s) [noise=true , address=45.91.116.5:4299 , piece_id=DQP4JZU4EXISXNWZ3E6NHAWTCFSH2GEPGBQS52T53S5DWQ6RV36Q, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.182796313 (532h8m51.053879359s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.182807064 (532h8m51.053890109s) [noise=true , address=77.90.31.18:4808 , piece_id=PFUWCIXEWD5AYBMTUSNDPV7DMCQ7GBHZTWIELN5EEUK5UFLSHY3A, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.183015981 (532h8m51.054099026s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 19:37:51.183025179 (532h8m51.054108224s) [noise=true , address=77.90.31.12:4589 , piece_id=2FBROGKYX2UUVMPYQ7IQXSOUGA4IG4PJGVBWQ4DXSYGMG66LPTCQ, seg_pos={0 0}, upload=466697]: Long tail cancellation due to optimal threshold Dec 27 19:37:51.183045598 (532h8m51.054128643s) [seg_pos={0 0}, upload=466697]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:37:51.183132633 (532h8m51.054215678s) [upload=466697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:37:51.183139386 (532h8m51.054222431s) [upload=466697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:37:51.477064154 (532h8m51.348147209s) [upload=466697]: Done uploading segments... Dec 27 19:37:51.477092258 (532h8m51.348175293s) [upload=466697]: Upload finished. err: <nil> Duration: 30.002975098s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.499909643s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:37:48 UTC) orbiter warning: context deadline exceeded Duration: 3.499937075s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)