Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.416224251s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:20:47 UTC) orbiter warning: context deadline exceeded Duration: 6.416254127s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.404891201s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:20:15 UTC) orbiter warning: context deadline exceeded Duration: 3.404922812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.....X.....XXX.X......................X..X.XX.X.X..X..X.] duration: 5.868151163s trace: https://jaeger.dev.storj.io/trace/458d0f6245f78b06 FAIL (2025-12-26 08:19:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA) context deadline exceeded ======LOGS====== Dec 26 08:18:52.134819319 (496h49m52.005902364s) [upload=436067]: Starting upload Dec 26 08:18:52.134906055 (496h49m52.0059891s) [upload=436067]: Done starting upload Dec 26 08:18:52.134979795 (496h49m52.00606283s) [upload=436067]: Uploading segments... Dec 26 08:18:52.135193111 (496h49m52.006276146s) [upload=436067]: Got next segment. Inline: false Dec 26 08:18:52.135222517 (496h49m52.006305552s) [upload=436067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:18:52.135227096 (496h49m52.006310131s) [upload=436067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:18:52.382805990 (496h49m52.253889065s) [seg_pos={0 0}, upload=436067]: Begin upload segment... Dec 26 08:18:52.382860142 (496h49m52.253943187s) [seg_pos={0 0}, upload=436067]: Done begin upload segment. Dec 26 08:18:52.382897624 (496h49m52.253980669s) [noise=true , address=145.79.227.45:4230 , piece_id=2ML7SHQ7XJRI3R6F5AJ6SZ4HW5KF5MLA23SM5VVOGNQUV4VDL3NQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.382945435 (496h49m52.25402847s) [noise=true , address=145.79.227.17:4970 , piece_id=YN6VSEO4P2RXL3BWTHHZJKETAP3BBEA2THMDJPEHB62APJ6PD53A, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.383473030 (496h49m52.254556065s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.383903780 (496h49m52.254986815s) [noise=true , address=77.90.31.13:4629 , piece_id=MIZKIVIFAIWICTIOFYHPI5ABHNUQHWHDZNSWYPETKENGVSRGSPUQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.384356261 (496h49m52.255439306s) [noise=true , address=145.79.227.51:4409 , piece_id=WF6XZJG7KX7I2Z4RD4KIPDBAKUVUO7CFG5WX4LSK5SSC7JLZXRBA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.384773626 (496h49m52.255856671s) [noise=true , address=145.79.227.12:4767 , piece_id=KUAFPNTEJCOGQOSLR2U5GLJHKAYERYB4XIUQDRMYIDBTTXHOXHXQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.385218363 (496h49m52.256301398s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.385710881 (496h49m52.256793916s) [noise=true , address=145.79.227.42:4122 , piece_id=W24LQ3LDO6KVYR3P7KR3UY4IYRPI7G33LDK2XA3PYR6VNXYZGEMA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.386836825 (496h49m52.257919861s) [noise=true , address=145.79.227.41:4087 , piece_id=CK46LF4X6PZ3UM632FRNWVA65H7HQQGDK3RVFCQS3LSX2Y75HSHA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.387375611 (496h49m52.258458646s) [noise=true , address=77.90.31.16:4736 , piece_id=F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.388295623 (496h49m52.259378658s) [noise=true , address=145.79.227.37:4565 , piece_id=JU5JOBMNSUZ3BXJ7O5U6DW4FNCD4A5W7P2JLQGZU6RBEOGWCNLOQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.391014010 (496h49m52.262097045s) [noise=true , address=145.79.227.15:4876 , piece_id=MAOK3GSF5LIL4CMAWL5WBB4T36H345QJPE5DUNDCVGZKCFPNNEQQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.394846126 (496h49m52.265929172s) [noise=true , address=45.91.116.4:4939 , piece_id=UCG5MDXWS7B5PU2MRAGG7EYRSTG4KJ3SFLJZUZJTNN7NFSWBAIZQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.399866796 (496h49m52.270949842s) [noise=true , address=145.79.227.6:4258 , piece_id=PASXVXIG6XTYTPXYRPIB56C2C5DNDJ7H3EJ7PG7PEMTFXFAQDEDQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.401440884 (496h49m52.272523929s) [noise=true , address=145.79.227.38:4979 , piece_id=3XUAB6QG2KCGZH2OG3VHLGY6WPHKHFA5XXDDCDKNUDIDNYHY4KVA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.404991454 (496h49m52.27607451s) [noise=true , address=77.90.31.14:4664 , piece_id=WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:53.722345789 (496h49m53.593428834s) [noise=true , address=145.79.227.37:4565 , piece_id=JU5JOBMNSUZ3BXJ7O5U6DW4FNCD4A5W7P2JLQGZU6RBEOGWCNLOQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: storage node overloaded, request limit: 2000 Dec 26 08:18:54.122882432 (496h49m53.993965467s) [upload=436067]: Next returned nil segment Dec 26 08:18:54.122896939 (496h49m53.993979984s) [upload=436067]: Waiting for error group managing segments... Dec 26 08:18:54.396051093 (496h49m54.267134128s) [noise=true , address=145.79.227.45:4230 , piece_id=2ML7SHQ7XJRI3R6F5AJ6SZ4HW5KF5MLA23SM5VVOGNQUV4VDL3NQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.403055701 (496h49m54.274138736s) [noise=true , address=145.79.227.41:4087 , piece_id=CK46LF4X6PZ3UM632FRNWVA65H7HQQGDK3RVFCQS3LSX2Y75HSHA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.408596261 (496h49m54.279679306s) [noise=true , address=145.79.227.6:4258 , piece_id=PASXVXIG6XTYTPXYRPIB56C2C5DNDJ7H3EJ7PG7PEMTFXFAQDEDQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.409983382 (496h49m54.281066417s) [noise=true , address=145.79.227.38:4979 , piece_id=3XUAB6QG2KCGZH2OG3VHLGY6WPHKHFA5XXDDCDKNUDIDNYHY4KVA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.415846607 (496h49m54.286929643s) [noise=true , address=145.79.227.42:4122 , piece_id=W24LQ3LDO6KVYR3P7KR3UY4IYRPI7G33LDK2XA3PYR6VNXYZGEMA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.420956807 (496h49m54.292039852s) [noise=true , address=145.79.227.12:4767 , piece_id=KUAFPNTEJCOGQOSLR2U5GLJHKAYERYB4XIUQDRMYIDBTTXHOXHXQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.422031644 (496h49m54.293114679s) [noise=true , address=145.79.227.51:4409 , piece_id=WF6XZJG7KX7I2Z4RD4KIPDBAKUVUO7CFG5WX4LSK5SSC7JLZXRBA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.428515050 (496h49m54.299598085s) [noise=true , address=145.79.227.17:4970 , piece_id=YN6VSEO4P2RXL3BWTHHZJKETAP3BBEA2THMDJPEHB62APJ6PD53A, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.441698103 (496h49m54.312781148s) [noise=true , address=145.79.227.15:4876 , piece_id=MAOK3GSF5LIL4CMAWL5WBB4T36H345QJPE5DUNDCVGZKCFPNNEQQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.603821214 (496h49m54.474904249s) [noise=true , address=77.90.31.14:4664 , piece_id=WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.606761101 (496h49m54.477844146s) [noise=true , address=77.90.31.16:4736 , piece_id=F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.611521737 (496h49m54.482604772s) [noise=true , address=77.90.31.13:4629 , piece_id=MIZKIVIFAIWICTIOFYHPI5ABHNUQHWHDZNSWYPETKENGVSRGSPUQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.632211042 (496h49m54.503294077s) [noise=true , address=45.91.116.4:4939 , piece_id=UCG5MDXWS7B5PU2MRAGG7EYRSTG4KJ3SFLJZUZJTNN7NFSWBAIZQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.632233024 (496h49m54.50331606s) [seg_pos={0 0}, upload=436067]: Segment reached optimal threshold of 13 pieces. Dec 26 08:18:54.632510373 (496h49m54.503593418s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 08:18:54.632559345 (496h49m54.503642401s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Long tail cancellation due to optimal threshold Dec 26 08:18:54.632777912 (496h49m54.503860957s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 08:18:54.632792119 (496h49m54.503875154s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Long tail cancellation due to optimal threshold Dec 26 08:18:54.632815003 (496h49m54.503898048s) [seg_pos={0 0}, upload=436067]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:18:54.632934260 (496h49m54.504017295s) [upload=436067]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:18:54.632941122 (496h49m54.504024168s) [upload=436067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:18:54.925285408 (496h49m54.796368443s) [upload=436067]: Done uploading segments... Dec 26 08:18:54.925304103 (496h49m54.796387148s) [upload=436067]: Upload finished. err: <nil> Duration: 5.868183523s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.98700468s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:19:47 UTC) orbiter warning: context deadline exceeded Duration: 5.98703097s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.459976716s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:19:39 UTC) orbiter warning: context deadline exceeded Duration: 3.460001453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...X.....X.....XXX.X......................X..X.XX.X.X..X..X] duration: 30.002538367s trace: https://jaeger.dev.storj.io/trace/458d0f6245f78b06 FAIL (2025-12-26 08:19:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA) context deadline exceeded ======LOGS====== Dec 26 08:18:52.134819319 (496h49m52.005902364s) [upload=436067]: Starting upload Dec 26 08:18:52.134906055 (496h49m52.0059891s) [upload=436067]: Done starting upload Dec 26 08:18:52.134979795 (496h49m52.00606283s) [upload=436067]: Uploading segments... Dec 26 08:18:52.135193111 (496h49m52.006276146s) [upload=436067]: Got next segment. Inline: false Dec 26 08:18:52.135222517 (496h49m52.006305552s) [upload=436067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:18:52.135227096 (496h49m52.006310131s) [upload=436067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:18:52.382805990 (496h49m52.253889065s) [seg_pos={0 0}, upload=436067]: Begin upload segment... Dec 26 08:18:52.382860142 (496h49m52.253943187s) [seg_pos={0 0}, upload=436067]: Done begin upload segment. Dec 26 08:18:52.382897624 (496h49m52.253980669s) [noise=true , address=145.79.227.45:4230 , piece_id=2ML7SHQ7XJRI3R6F5AJ6SZ4HW5KF5MLA23SM5VVOGNQUV4VDL3NQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.382945435 (496h49m52.25402847s) [noise=true , address=145.79.227.17:4970 , piece_id=YN6VSEO4P2RXL3BWTHHZJKETAP3BBEA2THMDJPEHB62APJ6PD53A, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.383473030 (496h49m52.254556065s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.383903780 (496h49m52.254986815s) [noise=true , address=77.90.31.13:4629 , piece_id=MIZKIVIFAIWICTIOFYHPI5ABHNUQHWHDZNSWYPETKENGVSRGSPUQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.384356261 (496h49m52.255439306s) [noise=true , address=145.79.227.51:4409 , piece_id=WF6XZJG7KX7I2Z4RD4KIPDBAKUVUO7CFG5WX4LSK5SSC7JLZXRBA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.384773626 (496h49m52.255856671s) [noise=true , address=145.79.227.12:4767 , piece_id=KUAFPNTEJCOGQOSLR2U5GLJHKAYERYB4XIUQDRMYIDBTTXHOXHXQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.385218363 (496h49m52.256301398s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.385710881 (496h49m52.256793916s) [noise=true , address=145.79.227.42:4122 , piece_id=W24LQ3LDO6KVYR3P7KR3UY4IYRPI7G33LDK2XA3PYR6VNXYZGEMA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.386836825 (496h49m52.257919861s) [noise=true , address=145.79.227.41:4087 , piece_id=CK46LF4X6PZ3UM632FRNWVA65H7HQQGDK3RVFCQS3LSX2Y75HSHA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.387375611 (496h49m52.258458646s) [noise=true , address=77.90.31.16:4736 , piece_id=F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.388295623 (496h49m52.259378658s) [noise=true , address=145.79.227.37:4565 , piece_id=JU5JOBMNSUZ3BXJ7O5U6DW4FNCD4A5W7P2JLQGZU6RBEOGWCNLOQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.391014010 (496h49m52.262097045s) [noise=true , address=145.79.227.15:4876 , piece_id=MAOK3GSF5LIL4CMAWL5WBB4T36H345QJPE5DUNDCVGZKCFPNNEQQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.394846126 (496h49m52.265929172s) [noise=true , address=45.91.116.4:4939 , piece_id=UCG5MDXWS7B5PU2MRAGG7EYRSTG4KJ3SFLJZUZJTNN7NFSWBAIZQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.399866796 (496h49m52.270949842s) [noise=true , address=145.79.227.6:4258 , piece_id=PASXVXIG6XTYTPXYRPIB56C2C5DNDJ7H3EJ7PG7PEMTFXFAQDEDQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.401440884 (496h49m52.272523929s) [noise=true , address=145.79.227.38:4979 , piece_id=3XUAB6QG2KCGZH2OG3VHLGY6WPHKHFA5XXDDCDKNUDIDNYHY4KVA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.404991454 (496h49m52.27607451s) [noise=true , address=77.90.31.14:4664 , piece_id=WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:53.722345789 (496h49m53.593428834s) [noise=true , address=145.79.227.37:4565 , piece_id=JU5JOBMNSUZ3BXJ7O5U6DW4FNCD4A5W7P2JLQGZU6RBEOGWCNLOQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: storage node overloaded, request limit: 2000 Dec 26 08:18:54.122882432 (496h49m53.993965467s) [upload=436067]: Next returned nil segment Dec 26 08:18:54.122896939 (496h49m53.993979984s) [upload=436067]: Waiting for error group managing segments... Dec 26 08:18:54.396051093 (496h49m54.267134128s) [noise=true , address=145.79.227.45:4230 , piece_id=2ML7SHQ7XJRI3R6F5AJ6SZ4HW5KF5MLA23SM5VVOGNQUV4VDL3NQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.403055701 (496h49m54.274138736s) [noise=true , address=145.79.227.41:4087 , piece_id=CK46LF4X6PZ3UM632FRNWVA65H7HQQGDK3RVFCQS3LSX2Y75HSHA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.408596261 (496h49m54.279679306s) [noise=true , address=145.79.227.6:4258 , piece_id=PASXVXIG6XTYTPXYRPIB56C2C5DNDJ7H3EJ7PG7PEMTFXFAQDEDQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.409983382 (496h49m54.281066417s) [noise=true , address=145.79.227.38:4979 , piece_id=3XUAB6QG2KCGZH2OG3VHLGY6WPHKHFA5XXDDCDKNUDIDNYHY4KVA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.415846607 (496h49m54.286929643s) [noise=true , address=145.79.227.42:4122 , piece_id=W24LQ3LDO6KVYR3P7KR3UY4IYRPI7G33LDK2XA3PYR6VNXYZGEMA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.420956807 (496h49m54.292039852s) [noise=true , address=145.79.227.12:4767 , piece_id=KUAFPNTEJCOGQOSLR2U5GLJHKAYERYB4XIUQDRMYIDBTTXHOXHXQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.422031644 (496h49m54.293114679s) [noise=true , address=145.79.227.51:4409 , piece_id=WF6XZJG7KX7I2Z4RD4KIPDBAKUVUO7CFG5WX4LSK5SSC7JLZXRBA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.428515050 (496h49m54.299598085s) [noise=true , address=145.79.227.17:4970 , piece_id=YN6VSEO4P2RXL3BWTHHZJKETAP3BBEA2THMDJPEHB62APJ6PD53A, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.441698103 (496h49m54.312781148s) [noise=true , address=145.79.227.15:4876 , piece_id=MAOK3GSF5LIL4CMAWL5WBB4T36H345QJPE5DUNDCVGZKCFPNNEQQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.603821214 (496h49m54.474904249s) [noise=true , address=77.90.31.14:4664 , piece_id=WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.606761101 (496h49m54.477844146s) [noise=true , address=77.90.31.16:4736 , piece_id=F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.611521737 (496h49m54.482604772s) [noise=true , address=77.90.31.13:4629 , piece_id=MIZKIVIFAIWICTIOFYHPI5ABHNUQHWHDZNSWYPETKENGVSRGSPUQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.632211042 (496h49m54.503294077s) [noise=true , address=45.91.116.4:4939 , piece_id=UCG5MDXWS7B5PU2MRAGG7EYRSTG4KJ3SFLJZUZJTNN7NFSWBAIZQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.632233024 (496h49m54.50331606s) [seg_pos={0 0}, upload=436067]: Segment reached optimal threshold of 13 pieces. Dec 26 08:18:54.632510373 (496h49m54.503593418s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 08:18:54.632559345 (496h49m54.503642401s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Long tail cancellation due to optimal threshold Dec 26 08:18:54.632777912 (496h49m54.503860957s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 08:18:54.632792119 (496h49m54.503875154s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Long tail cancellation due to optimal threshold Dec 26 08:18:54.632815003 (496h49m54.503898048s) [seg_pos={0 0}, upload=436067]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:18:54.632934260 (496h49m54.504017295s) [upload=436067]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:18:54.632941122 (496h49m54.504024168s) [upload=436067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:18:54.925285408 (496h49m54.796368443s) [upload=436067]: Done uploading segments... Dec 26 08:18:54.925304103 (496h49m54.796387148s) [upload=436067]: Upload finished. err: <nil> Duration: 30.002559457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.564987775s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:18:55 UTC) orbiter warning: context deadline exceeded Duration: 3.565012743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.145924722s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:18:52 UTC) orbiter warning: context deadline exceeded Duration: 6.145951213s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.356809146s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:18:26 UTC) orbiter warning: context deadline exceeded Duration: 3.356834504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.970244585s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:18:24 UTC) orbiter warning: context deadline exceeded Duration: 5.970293818s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.936524471s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:18:03 UTC) orbiter warning: context deadline exceeded Duration: 6.936569246s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...X.....X.....XXX.X......................X..X.XX.X.X..X..] duration: 6.01618937s trace: https://jaeger.dev.storj.io/trace/44b8d37bbaf1cbd0 FAIL (2025-12-26 08:17:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA) context deadline exceeded ======LOGS====== 82 (496h47m3.620943537s) [upload=436031]: Uploading segments... Dec 26 08:16:03.750170222 (496h47m3.621253337s) [upload=436031]: Got next segment. Inline: false Dec 26 08:16:03.750217301 (496h47m3.621300336s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:16:03.750225136 (496h47m3.621308181s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:16:03.996028027 (496h47m3.867111082s) [seg_pos={0 0}, upload=436031]: Begin upload segment... Dec 26 08:16:03.996201377 (496h47m3.867284422s) [seg_pos={0 0}, upload=436031]: Done begin upload segment. Dec 26 08:16:03.996260529 (496h47m3.867343574s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996356201 (496h47m3.867439246s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996987703 (496h47m3.868070749s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.997416861 (496h47m3.868499906s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998071046 (496h47m3.869154091s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998678824 (496h47m3.869761869s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999124643 (496h47m3.870207688s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999669781 (496h47m3.870752826s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.000416904 (496h47m3.871499959s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002163089 (496h47m3.873246134s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002923166 (496h47m3.874006221s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.003684706 (496h47m3.874767771s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004377015 (496h47m3.87546007s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004494839 (496h47m3.875577884s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.005322004 (496h47m3.876405049s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.006274337 (496h47m3.877357372s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:05.709486098 (496h47m5.580569143s) [upload=436031]: Next returned nil segment Dec 26 08:16:05.709504011 (496h47m5.580587046s) [upload=436031]: Waiting for error group managing segments... Dec 26 08:16:05.997101825 (496h47m5.86818486s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.002826386 (496h47m5.873909431s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.003169179 (496h47m5.874252244s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.015164881 (496h47m5.886247916s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.017734023 (496h47m5.888817068s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.018932005 (496h47m5.89001504s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.020254483 (496h47m5.891337528s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.024683606 (496h47m5.895766651s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036334721 (496h47m5.907417756s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036433650 (496h47m5.907516695s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.251058020 (496h47m6.122141065s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.355066080 (496h47m6.226149115s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376711365 (496h47m6.2477944s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376731875 (496h47m6.24781491s) [seg_pos={0 0}, upload=436031]: Segment reached optimal threshold of 13 pieces. Dec 26 08:16:06.376924802 (496h47m6.248007837s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection Dec 26 08:16:06.376932617 (496h47m6.248015652s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.376963205 (496h47m6.24804625s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 08:16:06.376973816 (496h47m6.248056861s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377029190 (496h47m6.248112236s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection Dec 26 08:16:06.377035894 (496h47m6.248118929s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377050131 (496h47m6.248133166s) [seg_pos={0 0}, upload=436031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:16:06.377121917 (496h47m6.248204952s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:16:06.377127738 (496h47m6.248210773s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:16:06.671006954 (496h47m6.542089989s) [upload=436031]: Done uploading segments... Dec 26 08:16:06.671028706 (496h47m6.542111741s) [upload=436031]: Upload finished. err: <nil> Duration: 6.016218046s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.883125384s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:17:41 UTC) orbiter warning: context deadline exceeded Duration: 5.883150892s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.461669739s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:17:34 UTC) orbiter warning: context deadline exceeded Duration: 3.461700296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...X.....X.....XXX.X......................X..X.XX.X.X..X.] duration: 6.072190278s trace: https://jaeger.dev.storj.io/trace/44b8d37bbaf1cbd0 FAIL (2025-12-26 08:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA) context deadline exceeded ======LOGS====== 82 (496h47m3.620943537s) [upload=436031]: Uploading segments... Dec 26 08:16:03.750170222 (496h47m3.621253337s) [upload=436031]: Got next segment. Inline: false Dec 26 08:16:03.750217301 (496h47m3.621300336s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:16:03.750225136 (496h47m3.621308181s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:16:03.996028027 (496h47m3.867111082s) [seg_pos={0 0}, upload=436031]: Begin upload segment... Dec 26 08:16:03.996201377 (496h47m3.867284422s) [seg_pos={0 0}, upload=436031]: Done begin upload segment. Dec 26 08:16:03.996260529 (496h47m3.867343574s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996356201 (496h47m3.867439246s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996987703 (496h47m3.868070749s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.997416861 (496h47m3.868499906s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998071046 (496h47m3.869154091s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998678824 (496h47m3.869761869s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999124643 (496h47m3.870207688s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999669781 (496h47m3.870752826s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.000416904 (496h47m3.871499959s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002163089 (496h47m3.873246134s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002923166 (496h47m3.874006221s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.003684706 (496h47m3.874767771s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004377015 (496h47m3.87546007s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004494839 (496h47m3.875577884s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.005322004 (496h47m3.876405049s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.006274337 (496h47m3.877357372s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:05.709486098 (496h47m5.580569143s) [upload=436031]: Next returned nil segment Dec 26 08:16:05.709504011 (496h47m5.580587046s) [upload=436031]: Waiting for error group managing segments... Dec 26 08:16:05.997101825 (496h47m5.86818486s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.002826386 (496h47m5.873909431s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.003169179 (496h47m5.874252244s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.015164881 (496h47m5.886247916s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.017734023 (496h47m5.888817068s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.018932005 (496h47m5.89001504s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.020254483 (496h47m5.891337528s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.024683606 (496h47m5.895766651s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036334721 (496h47m5.907417756s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036433650 (496h47m5.907516695s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.251058020 (496h47m6.122141065s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.355066080 (496h47m6.226149115s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376711365 (496h47m6.2477944s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376731875 (496h47m6.24781491s) [seg_pos={0 0}, upload=436031]: Segment reached optimal threshold of 13 pieces. Dec 26 08:16:06.376924802 (496h47m6.248007837s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection Dec 26 08:16:06.376932617 (496h47m6.248015652s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.376963205 (496h47m6.24804625s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 08:16:06.376973816 (496h47m6.248056861s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377029190 (496h47m6.248112236s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection Dec 26 08:16:06.377035894 (496h47m6.248118929s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377050131 (496h47m6.248133166s) [seg_pos={0 0}, upload=436031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:16:06.377121917 (496h47m6.248204952s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:16:06.377127738 (496h47m6.248210773s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:16:06.671006954 (496h47m6.542089989s) [upload=436031]: Done uploading segments... Dec 26 08:16:06.671028706 (496h47m6.542111741s) [upload=436031]: Upload finished. err: <nil> Duration: 6.072209555s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.471084627s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:17:05 UTC) orbiter warning: context deadline exceeded Duration: 3.471153538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X...X.....X.....XXX.X......................X..X.XX.X.X..X] duration: 30.002827777s trace: https://jaeger.dev.storj.io/trace/44b8d37bbaf1cbd0 FAIL (2025-12-26 08:16:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA) context deadline exceeded ======LOGS====== 82 (496h47m3.620943537s) [upload=436031]: Uploading segments... Dec 26 08:16:03.750170222 (496h47m3.621253337s) [upload=436031]: Got next segment. Inline: false Dec 26 08:16:03.750217301 (496h47m3.621300336s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:16:03.750225136 (496h47m3.621308181s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:16:03.996028027 (496h47m3.867111082s) [seg_pos={0 0}, upload=436031]: Begin upload segment... Dec 26 08:16:03.996201377 (496h47m3.867284422s) [seg_pos={0 0}, upload=436031]: Done begin upload segment. Dec 26 08:16:03.996260529 (496h47m3.867343574s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996356201 (496h47m3.867439246s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996987703 (496h47m3.868070749s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.997416861 (496h47m3.868499906s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998071046 (496h47m3.869154091s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998678824 (496h47m3.869761869s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999124643 (496h47m3.870207688s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999669781 (496h47m3.870752826s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.000416904 (496h47m3.871499959s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002163089 (496h47m3.873246134s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002923166 (496h47m3.874006221s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.003684706 (496h47m3.874767771s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004377015 (496h47m3.87546007s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004494839 (496h47m3.875577884s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.005322004 (496h47m3.876405049s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.006274337 (496h47m3.877357372s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:05.709486098 (496h47m5.580569143s) [upload=436031]: Next returned nil segment Dec 26 08:16:05.709504011 (496h47m5.580587046s) [upload=436031]: Waiting for error group managing segments... Dec 26 08:16:05.997101825 (496h47m5.86818486s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.002826386 (496h47m5.873909431s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.003169179 (496h47m5.874252244s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.015164881 (496h47m5.886247916s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.017734023 (496h47m5.888817068s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.018932005 (496h47m5.89001504s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.020254483 (496h47m5.891337528s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.024683606 (496h47m5.895766651s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036334721 (496h47m5.907417756s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036433650 (496h47m5.907516695s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.251058020 (496h47m6.122141065s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.355066080 (496h47m6.226149115s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376711365 (496h47m6.2477944s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376731875 (496h47m6.24781491s) [seg_pos={0 0}, upload=436031]: Segment reached optimal threshold of 13 pieces. Dec 26 08:16:06.376924802 (496h47m6.248007837s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection Dec 26 08:16:06.376932617 (496h47m6.248015652s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.376963205 (496h47m6.24804625s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 08:16:06.376973816 (496h47m6.248056861s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377029190 (496h47m6.248112236s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection Dec 26 08:16:06.377035894 (496h47m6.248118929s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377050131 (496h47m6.248133166s) [seg_pos={0 0}, upload=436031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:16:06.377121917 (496h47m6.248204952s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:16:06.377127738 (496h47m6.248210773s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:16:06.671006954 (496h47m6.542089989s) [upload=436031]: Done uploading segments... Dec 26 08:16:06.671028706 (496h47m6.542111741s) [upload=436031]: Upload finished. err: <nil> Duration: 30.00288253s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.920008137s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:16:33 UTC) orbiter warning: context deadline exceeded Duration: 5.920032903s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.471471601s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:16:28 UTC) orbiter warning: context deadline exceeded Duration: 3.47149687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.915534257s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:16:16 UTC) orbiter warning: context deadline exceeded Duration: 5.915568543s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X...X.....X.....XXX.X......................X..X.XX.X.X..] duration: 5.694763623s trace: https://jaeger.dev.storj.io/trace/5418ee64ef539afc FAIL (2025-12-26 08:15:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ) context deadline exceeded ======LOGS====== Dec 26 08:13:56.387591062 (496h44m56.258674097s) [upload=435998]: Starting upload Dec 26 08:13:56.387645205 (496h44m56.25872824s) [upload=435998]: Done starting upload Dec 26 08:13:56.387743602 (496h44m56.258826637s) [upload=435998]: Uploading segments... Dec 26 08:13:56.387897605 (496h44m56.258980671s) [upload=435998]: Got next segment. Inline: false Dec 26 08:13:56.387939104 (496h44m56.259022159s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:13:56.387946799 (496h44m56.259029844s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:13:56.632308335 (496h44m56.50339137s) [seg_pos={0 0}, upload=435998]: Begin upload segment... Dec 26 08:13:56.632367236 (496h44m56.503450271s) [seg_pos={0 0}, upload=435998]: Done begin upload segment. Dec 26 08:13:56.632416010 (496h44m56.503499045s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.632480863 (496h44m56.503563898s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633103479 (496h44m56.504186524s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633422737 (496h44m56.504505782s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633910335 (496h44m56.50499337s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634257015 (496h44m56.50534006s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634560162 (496h44m56.505643197s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634808494 (496h44m56.50589154s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634898035 (496h44m56.50598107s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635211773 (496h44m56.506294818s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635243203 (496h44m56.506326238s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635634247 (496h44m56.506717292s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636007919 (496h44m56.507090954s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636378253 (496h44m56.507461299s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636760842 (496h44m56.507843887s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.639144031 (496h44m56.510227066s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:58.350452979 (496h44m58.221536024s) [upload=435998]: Next returned nil segment Dec 26 08:13:58.350510920 (496h44m58.221593975s) [upload=435998]: Waiting for error group managing segments... Dec 26 08:13:58.636874774 (496h44m58.50795782s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.645244031 (496h44m58.516327076s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.648582838 (496h44m58.519665883s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.650931201 (496h44m58.522014236s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.652815889 (496h44m58.523898924s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.654005154 (496h44m58.525088189s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.660084860 (496h44m58.531167895s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.667454473 (496h44m58.538537508s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.674669892 (496h44m58.545752927s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.676972116 (496h44m58.548055151s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.972357689 (496h44m58.843440725s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.976407091 (496h44m58.847490136s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995564359 (496h44m58.866647394s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995585260 (496h44m58.866668305s) [seg_pos={0 0}, upload=435998]: Segment reached optimal threshold of 13 pieces. Dec 26 08:13:58.995849242 (496h44m58.866932307s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995859150 (496h44m58.866942185s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.995890881 (496h44m58.866973946s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995900249 (496h44m58.866983284s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996085993 (496h44m58.867169048s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.996094178 (496h44m58.867177213s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996114267 (496h44m58.867197302s) [seg_pos={0 0}, upload=435998]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:13:58.996199899 (496h44m58.867282934s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:13:58.996207173 (496h44m58.867290218s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:13:59.287409842 (496h44m59.158492887s) [upload=435998]: Done uploading segments... Dec 26 08:13:59.287430241 (496h44m59.158513276s) [upload=435998]: Upload finished. err: <nil> Duration: 5.694786106s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.485047854s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:15:36 UTC) orbiter warning: context deadline exceeded Duration: 3.485076608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.904032426s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:15:13 UTC) orbiter warning: context deadline exceeded Duration: 5.904101166s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.432165668s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:15:11 UTC) orbiter warning: context deadline exceeded Duration: 3.432186998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X....X...X.....X.....XXX.X......................X..X.XX.X.X.] duration: 5.999915594s trace: https://jaeger.dev.storj.io/trace/5418ee64ef539afc FAIL (2025-12-26 08:15:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ) context deadline exceeded ======LOGS====== Dec 26 08:13:56.387591062 (496h44m56.258674097s) [upload=435998]: Starting upload Dec 26 08:13:56.387645205 (496h44m56.25872824s) [upload=435998]: Done starting upload Dec 26 08:13:56.387743602 (496h44m56.258826637s) [upload=435998]: Uploading segments... Dec 26 08:13:56.387897605 (496h44m56.258980671s) [upload=435998]: Got next segment. Inline: false Dec 26 08:13:56.387939104 (496h44m56.259022159s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:13:56.387946799 (496h44m56.259029844s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:13:56.632308335 (496h44m56.50339137s) [seg_pos={0 0}, upload=435998]: Begin upload segment... Dec 26 08:13:56.632367236 (496h44m56.503450271s) [seg_pos={0 0}, upload=435998]: Done begin upload segment. Dec 26 08:13:56.632416010 (496h44m56.503499045s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.632480863 (496h44m56.503563898s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633103479 (496h44m56.504186524s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633422737 (496h44m56.504505782s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633910335 (496h44m56.50499337s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634257015 (496h44m56.50534006s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634560162 (496h44m56.505643197s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634808494 (496h44m56.50589154s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634898035 (496h44m56.50598107s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635211773 (496h44m56.506294818s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635243203 (496h44m56.506326238s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635634247 (496h44m56.506717292s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636007919 (496h44m56.507090954s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636378253 (496h44m56.507461299s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636760842 (496h44m56.507843887s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.639144031 (496h44m56.510227066s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:58.350452979 (496h44m58.221536024s) [upload=435998]: Next returned nil segment Dec 26 08:13:58.350510920 (496h44m58.221593975s) [upload=435998]: Waiting for error group managing segments... Dec 26 08:13:58.636874774 (496h44m58.50795782s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.645244031 (496h44m58.516327076s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.648582838 (496h44m58.519665883s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.650931201 (496h44m58.522014236s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.652815889 (496h44m58.523898924s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.654005154 (496h44m58.525088189s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.660084860 (496h44m58.531167895s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.667454473 (496h44m58.538537508s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.674669892 (496h44m58.545752927s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.676972116 (496h44m58.548055151s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.972357689 (496h44m58.843440725s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.976407091 (496h44m58.847490136s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995564359 (496h44m58.866647394s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995585260 (496h44m58.866668305s) [seg_pos={0 0}, upload=435998]: Segment reached optimal threshold of 13 pieces. Dec 26 08:13:58.995849242 (496h44m58.866932307s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995859150 (496h44m58.866942185s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.995890881 (496h44m58.866973946s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995900249 (496h44m58.866983284s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996085993 (496h44m58.867169048s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.996094178 (496h44m58.867177213s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996114267 (496h44m58.867197302s) [seg_pos={0 0}, upload=435998]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:13:58.996199899 (496h44m58.867282934s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:13:58.996207173 (496h44m58.867290218s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:13:59.287409842 (496h44m59.158492887s) [upload=435998]: Done uploading segments... Dec 26 08:13:59.287430241 (496h44m59.158513276s) [upload=435998]: Upload finished. err: <nil> Duration: 5.999943877s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.154531408s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:14:31 UTC) orbiter warning: context deadline exceeded Duration: 6.154576193s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX....X...X.....X.....XXX.X......................X..X.XX.X.X] duration: 30.003453606s trace: https://jaeger.dev.storj.io/trace/5418ee64ef539afc FAIL (2025-12-26 08:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ) context deadline exceeded ======LOGS====== Dec 26 08:13:56.387591062 (496h44m56.258674097s) [upload=435998]: Starting upload Dec 26 08:13:56.387645205 (496h44m56.25872824s) [upload=435998]: Done starting upload Dec 26 08:13:56.387743602 (496h44m56.258826637s) [upload=435998]: Uploading segments... Dec 26 08:13:56.387897605 (496h44m56.258980671s) [upload=435998]: Got next segment. Inline: false Dec 26 08:13:56.387939104 (496h44m56.259022159s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:13:56.387946799 (496h44m56.259029844s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:13:56.632308335 (496h44m56.50339137s) [seg_pos={0 0}, upload=435998]: Begin upload segment... Dec 26 08:13:56.632367236 (496h44m56.503450271s) [seg_pos={0 0}, upload=435998]: Done begin upload segment. Dec 26 08:13:56.632416010 (496h44m56.503499045s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.632480863 (496h44m56.503563898s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633103479 (496h44m56.504186524s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633422737 (496h44m56.504505782s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633910335 (496h44m56.50499337s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634257015 (496h44m56.50534006s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634560162 (496h44m56.505643197s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634808494 (496h44m56.50589154s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634898035 (496h44m56.50598107s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635211773 (496h44m56.506294818s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635243203 (496h44m56.506326238s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635634247 (496h44m56.506717292s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636007919 (496h44m56.507090954s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636378253 (496h44m56.507461299s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636760842 (496h44m56.507843887s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.639144031 (496h44m56.510227066s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:58.350452979 (496h44m58.221536024s) [upload=435998]: Next returned nil segment Dec 26 08:13:58.350510920 (496h44m58.221593975s) [upload=435998]: Waiting for error group managing segments... Dec 26 08:13:58.636874774 (496h44m58.50795782s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.645244031 (496h44m58.516327076s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.648582838 (496h44m58.519665883s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.650931201 (496h44m58.522014236s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.652815889 (496h44m58.523898924s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.654005154 (496h44m58.525088189s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.660084860 (496h44m58.531167895s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.667454473 (496h44m58.538537508s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.674669892 (496h44m58.545752927s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.676972116 (496h44m58.548055151s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.972357689 (496h44m58.843440725s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.976407091 (496h44m58.847490136s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995564359 (496h44m58.866647394s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995585260 (496h44m58.866668305s) [seg_pos={0 0}, upload=435998]: Segment reached optimal threshold of 13 pieces. Dec 26 08:13:58.995849242 (496h44m58.866932307s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995859150 (496h44m58.866942185s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.995890881 (496h44m58.866973946s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995900249 (496h44m58.866983284s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996085993 (496h44m58.867169048s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.996094178 (496h44m58.867177213s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996114267 (496h44m58.867197302s) [seg_pos={0 0}, upload=435998]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:13:58.996199899 (496h44m58.867282934s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:13:58.996207173 (496h44m58.867290218s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:13:59.287409842 (496h44m59.158492887s) [upload=435998]: Done uploading segments... Dec 26 08:13:59.287430241 (496h44m59.158513276s) [upload=435998]: Upload finished. err: <nil> Duration: 30.003476721s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.974683161s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:14:11 UTC) orbiter warning: context deadline exceeded Duration: 3.974759267s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 9.840040133s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:14:00 UTC) orbiter warning: context deadline exceeded Duration: 9.840122488s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.290595178s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:13:35 UTC) orbiter warning: context deadline exceeded Duration: 6.290619934s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.369206274s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:13:25 UTC) orbiter warning: context deadline exceeded Duration: 3.369230479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX....X...X.....X.....XXX.X......................X..X.XX.X.] duration: 6.088908241s trace: https://jaeger.dev.storj.io/trace/17d9d640ae0e3b3c FAIL (2025-12-26 08:12:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:11:43.193456026 (496h42m43.064539071s) [upload=435966]: Starting upload Dec 26 08:11:43.193520309 (496h42m43.064603344s) [upload=435966]: Done starting upload Dec 26 08:11:43.193652050 (496h42m43.064735105s) [upload=435966]: Uploading segments... Dec 26 08:11:43.193724478 (496h42m43.064807523s) [upload=435966]: Got next segment. Inline: false Dec 26 08:11:43.193821351 (496h42m43.064904386s) [upload=435966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:11:43.193849375 (496h42m43.06493243s) [upload=435966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:11:43.441602889 (496h42m43.312685924s) [seg_pos={0 0}, upload=435966]: Begin upload segment... Dec 26 08:11:43.441667572 (496h42m43.312750617s) [seg_pos={0 0}, upload=435966]: Done begin upload segment. Dec 26 08:11:43.441694123 (496h42m43.312777168s) [noise=true , address=145.79.227.36:4517 , piece_id=XF5NYT3EQGEPUADYXICF6TXGKVTOVK5C2JURNBYLC3N5EJHLEQIQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.441718871 (496h42m43.312801906s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442435905 (496h42m43.31351894s) [noise=true , address=145.79.227.19:4078 , piece_id=D5D4NF4U5OU3VJDFF4UNRDUYN53NAWFAOXE4IYYXT3VNRX32BWXQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442444543 (496h42m43.313527588s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442955134 (496h42m43.314038179s) [noise=true , address=145.79.227.41:4086 , piece_id=BH3IWHNWFDETI7E6BOIQRYXFYTXPOQSGYLFSQ77UO67NWQQIARRA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443024216 (496h42m43.314107261s) [noise=true , address=145.79.227.6:4257 , piece_id=PKSECFXIUAVZ5UWMMUHOWAWCFW7XUEGQJB6ZARVY3Q5LP4MNBGZQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443372629 (496h42m43.314455674s) [noise=true , address=145.79.227.3:4150 , piece_id=LKKAAOJXZ6EJLTRDZKE5EFTAD4IFZORFILMMAVQORMKU7B2JO5YA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443532684 (496h42m43.314615729s) [noise=true , address=145.79.227.51:4410 , piece_id=VB7VSJNUWT6LWOHL5NM37VDAOA4L2B7OTX3IF4LSNJWV5WRSQEGA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443967452 (496h42m43.315050497s) [noise=true , address=77.90.31.11:4556 , piece_id=TOFFKBK4XGH5Q2JKI37TQFWJMQY6QM46CGIFBLW5RS3P2JJAUQRA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444374397 (496h42m43.315457442s) [noise=true , address=145.79.227.52:4446 , piece_id=EG3ICHGG3RZBPCUELATHMFEXWNHHXTICCSCSFUCI67O5QN33PTEQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444704516 (496h42m43.315787551s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444924875 (496h42m43.31600792s) [noise=true , address=145.79.227.9:4367 , piece_id=3K2C6ZGHLYOHWLUQYFLLDHJXJB5H6BX4XOS67MICU3XM7BUGWLNA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.445350766 (496h42m43.316433811s) [noise=true , address=145.79.227.13:4803 , piece_id=JEW25O7EGLEYAADLG33ASB2O2JKVM5DCGZBGFA4A3NKWT33AJQKQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.445705341 (496h42m43.316788376s) [noise=true , address=145.79.227.16:4910 , piece_id=25WEOZ5MZLUI7OSJPAKWCQOMYUCO6RTP7P65W5U7FSBHTSDTQGUA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.446028307 (496h42m43.317111342s) [noise=true , address=45.91.116.6:4976 , piece_id=YMZNPO55XMKGDL6RNFHFOSTVGRYG4BZLF5Z4HNHXKKWLR2X4CHXQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.446432155 (496h42m43.3175152s) [noise=true , address=145.79.227.38:4979 , piece_id=C2EG7623HRJLTJW2QR4RVIOJ4JKPLRZM2NJHWDC7FNC32SX7RPKQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:45.153862759 (496h42m45.024945794s) [upload=435966]: Next returned nil segment Dec 26 08:11:45.153886234 (496h42m45.024969269s) [upload=435966]: Waiting for error group managing segments... Dec 26 08:11:45.448693466 (496h42m45.319776501s) [noise=true , address=145.79.227.51:4410 , piece_id=VB7VSJNUWT6LWOHL5NM37VDAOA4L2B7OTX3IF4LSNJWV5WRSQEGA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.452290074 (496h42m45.323373109s) [noise=true , address=145.79.227.36:4517 , piece_id=XF5NYT3EQGEPUADYXICF6TXGKVTOVK5C2JURNBYLC3N5EJHLEQIQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.453739825 (496h42m45.32482287s) [noise=true , address=145.79.227.38:4979 , piece_id=C2EG7623HRJLTJW2QR4RVIOJ4JKPLRZM2NJHWDC7FNC32SX7RPKQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.454108528 (496h42m45.325191563s) [noise=true , address=145.79.227.9:4367 , piece_id=3K2C6ZGHLYOHWLUQYFLLDHJXJB5H6BX4XOS67MICU3XM7BUGWLNA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.456227342 (496h42m45.327310377s) [noise=true , address=145.79.227.3:4150 , piece_id=LKKAAOJXZ6EJLTRDZKE5EFTAD4IFZORFILMMAVQORMKU7B2JO5YA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.457784807 (496h42m45.328867842s) [noise=true , address=145.79.227.13:4803 , piece_id=JEW25O7EGLEYAADLG33ASB2O2JKVM5DCGZBGFA4A3NKWT33AJQKQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.471772232 (496h42m45.342855277s) [noise=true , address=145.79.227.6:4257 , piece_id=PKSECFXIUAVZ5UWMMUHOWAWCFW7XUEGQJB6ZARVY3Q5LP4MNBGZQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.472312932 (496h42m45.343395977s) [noise=true , address=145.79.227.16:4910 , piece_id=25WEOZ5MZLUI7OSJPAKWCQOMYUCO6RTP7P65W5U7FSBHTSDTQGUA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.477544674 (496h42m45.348627709s) [noise=true , address=145.79.227.41:4086 , piece_id=BH3IWHNWFDETI7E6BOIQRYXFYTXPOQSGYLFSQ77UO67NWQQIARRA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.491847869 (496h42m45.362930904s) [noise=true , address=145.79.227.52:4446 , piece_id=EG3ICHGG3RZBPCUELATHMFEXWNHHXTICCSCSFUCI67O5QN33PTEQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.501135975 (496h42m45.37221902s) [noise=true , address=145.79.227.19:4078 , piece_id=D5D4NF4U5OU3VJDFF4UNRDUYN53NAWFAOXE4IYYXT3VNRX32BWXQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.608631485 (496h42m45.47971453s) [noise=true , address=77.90.31.11:4556 , piece_id=TOFFKBK4XGH5Q2JKI37TQFWJMQY6QM46CGIFBLW5RS3P2JJAUQRA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.728710294 (496h42m45.599793339s) [noise=true , address=45.91.116.6:4976 , piece_id=YMZNPO55XMKGDL6RNFHFOSTVGRYG4BZLF5Z4HNHXKKWLR2X4CHXQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.728735372 (496h42m45.599818417s) [seg_pos={0 0}, upload=435966]: Segment reached optimal threshold of 13 pieces. Dec 26 08:11:45.729023932 (496h42m45.600106988s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:36580->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:11:45.729034102 (496h42m45.600117147s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729235996 (496h42m45.600319061s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 08:11:45.729246326 (496h42m45.600329371s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729268638 (496h42m45.600351673s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 08:11:45.729279359 (496h42m45.600362394s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729311299 (496h42m45.600394344s) [seg_pos={0 0}, upload=435966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:11:45.729417842 (496h42m45.600500887s) [upload=435966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:11:45.729425286 (496h42m45.600508331s) [upload=435966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:11:46.015994742 (496h42m45.887077777s) [upload=435966]: Done uploading segments... Dec 26 08:11:46.016026893 (496h42m45.887109928s) [upload=435966]: Upload finished. err: <nil> Duration: 6.088936714s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.384474116s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:12:42 UTC) orbiter warning: context deadline exceeded Duration: 3.384499053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.931032689s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:12:29 UTC) orbiter warning: context deadline exceeded Duration: 5.931055773s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XX....X...X.....X.....XXX.X......................X..X.XX.X] duration: 30.003328781s trace: https://jaeger.dev.storj.io/trace/17d9d640ae0e3b3c FAIL (2025-12-26 08:12:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:11:43.193456026 (496h42m43.064539071s) [upload=435966]: Starting upload Dec 26 08:11:43.193520309 (496h42m43.064603344s) [upload=435966]: Done starting upload Dec 26 08:11:43.193652050 (496h42m43.064735105s) [upload=435966]: Uploading segments... Dec 26 08:11:43.193724478 (496h42m43.064807523s) [upload=435966]: Got next segment. Inline: false Dec 26 08:11:43.193821351 (496h42m43.064904386s) [upload=435966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:11:43.193849375 (496h42m43.06493243s) [upload=435966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:11:43.441602889 (496h42m43.312685924s) [seg_pos={0 0}, upload=435966]: Begin upload segment... Dec 26 08:11:43.441667572 (496h42m43.312750617s) [seg_pos={0 0}, upload=435966]: Done begin upload segment. Dec 26 08:11:43.441694123 (496h42m43.312777168s) [noise=true , address=145.79.227.36:4517 , piece_id=XF5NYT3EQGEPUADYXICF6TXGKVTOVK5C2JURNBYLC3N5EJHLEQIQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.441718871 (496h42m43.312801906s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442435905 (496h42m43.31351894s) [noise=true , address=145.79.227.19:4078 , piece_id=D5D4NF4U5OU3VJDFF4UNRDUYN53NAWFAOXE4IYYXT3VNRX32BWXQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442444543 (496h42m43.313527588s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442955134 (496h42m43.314038179s) [noise=true , address=145.79.227.41:4086 , piece_id=BH3IWHNWFDETI7E6BOIQRYXFYTXPOQSGYLFSQ77UO67NWQQIARRA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443024216 (496h42m43.314107261s) [noise=true , address=145.79.227.6:4257 , piece_id=PKSECFXIUAVZ5UWMMUHOWAWCFW7XUEGQJB6ZARVY3Q5LP4MNBGZQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443372629 (496h42m43.314455674s) [noise=true , address=145.79.227.3:4150 , piece_id=LKKAAOJXZ6EJLTRDZKE5EFTAD4IFZORFILMMAVQORMKU7B2JO5YA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443532684 (496h42m43.314615729s) [noise=true , address=145.79.227.51:4410 , piece_id=VB7VSJNUWT6LWOHL5NM37VDAOA4L2B7OTX3IF4LSNJWV5WRSQEGA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443967452 (496h42m43.315050497s) [noise=true , address=77.90.31.11:4556 , piece_id=TOFFKBK4XGH5Q2JKI37TQFWJMQY6QM46CGIFBLW5RS3P2JJAUQRA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444374397 (496h42m43.315457442s) [noise=true , address=145.79.227.52:4446 , piece_id=EG3ICHGG3RZBPCUELATHMFEXWNHHXTICCSCSFUCI67O5QN33PTEQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444704516 (496h42m43.315787551s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444924875 (496h42m43.31600792s) [noise=true , address=145.79.227.9:4367 , piece_id=3K2C6ZGHLYOHWLUQYFLLDHJXJB5H6BX4XOS67MICU3XM7BUGWLNA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.445350766 (496h42m43.316433811s) [noise=true , address=145.79.227.13:4803 , piece_id=JEW25O7EGLEYAADLG33ASB2O2JKVM5DCGZBGFA4A3NKWT33AJQKQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.445705341 (496h42m43.316788376s) [noise=true , address=145.79.227.16:4910 , piece_id=25WEOZ5MZLUI7OSJPAKWCQOMYUCO6RTP7P65W5U7FSBHTSDTQGUA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.446028307 (496h42m43.317111342s) [noise=true , address=45.91.116.6:4976 , piece_id=YMZNPO55XMKGDL6RNFHFOSTVGRYG4BZLF5Z4HNHXKKWLR2X4CHXQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.446432155 (496h42m43.3175152s) [noise=true , address=145.79.227.38:4979 , piece_id=C2EG7623HRJLTJW2QR4RVIOJ4JKPLRZM2NJHWDC7FNC32SX7RPKQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:45.153862759 (496h42m45.024945794s) [upload=435966]: Next returned nil segment Dec 26 08:11:45.153886234 (496h42m45.024969269s) [upload=435966]: Waiting for error group managing segments... Dec 26 08:11:45.448693466 (496h42m45.319776501s) [noise=true , address=145.79.227.51:4410 , piece_id=VB7VSJNUWT6LWOHL5NM37VDAOA4L2B7OTX3IF4LSNJWV5WRSQEGA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.452290074 (496h42m45.323373109s) [noise=true , address=145.79.227.36:4517 , piece_id=XF5NYT3EQGEPUADYXICF6TXGKVTOVK5C2JURNBYLC3N5EJHLEQIQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.453739825 (496h42m45.32482287s) [noise=true , address=145.79.227.38:4979 , piece_id=C2EG7623HRJLTJW2QR4RVIOJ4JKPLRZM2NJHWDC7FNC32SX7RPKQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.454108528 (496h42m45.325191563s) [noise=true , address=145.79.227.9:4367 , piece_id=3K2C6ZGHLYOHWLUQYFLLDHJXJB5H6BX4XOS67MICU3XM7BUGWLNA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.456227342 (496h42m45.327310377s) [noise=true , address=145.79.227.3:4150 , piece_id=LKKAAOJXZ6EJLTRDZKE5EFTAD4IFZORFILMMAVQORMKU7B2JO5YA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.457784807 (496h42m45.328867842s) [noise=true , address=145.79.227.13:4803 , piece_id=JEW25O7EGLEYAADLG33ASB2O2JKVM5DCGZBGFA4A3NKWT33AJQKQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.471772232 (496h42m45.342855277s) [noise=true , address=145.79.227.6:4257 , piece_id=PKSECFXIUAVZ5UWMMUHOWAWCFW7XUEGQJB6ZARVY3Q5LP4MNBGZQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.472312932 (496h42m45.343395977s) [noise=true , address=145.79.227.16:4910 , piece_id=25WEOZ5MZLUI7OSJPAKWCQOMYUCO6RTP7P65W5U7FSBHTSDTQGUA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.477544674 (496h42m45.348627709s) [noise=true , address=145.79.227.41:4086 , piece_id=BH3IWHNWFDETI7E6BOIQRYXFYTXPOQSGYLFSQ77UO67NWQQIARRA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.491847869 (496h42m45.362930904s) [noise=true , address=145.79.227.52:4446 , piece_id=EG3ICHGG3RZBPCUELATHMFEXWNHHXTICCSCSFUCI67O5QN33PTEQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.501135975 (496h42m45.37221902s) [noise=true , address=145.79.227.19:4078 , piece_id=D5D4NF4U5OU3VJDFF4UNRDUYN53NAWFAOXE4IYYXT3VNRX32BWXQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.608631485 (496h42m45.47971453s) [noise=true , address=77.90.31.11:4556 , piece_id=TOFFKBK4XGH5Q2JKI37TQFWJMQY6QM46CGIFBLW5RS3P2JJAUQRA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.728710294 (496h42m45.599793339s) [noise=true , address=45.91.116.6:4976 , piece_id=YMZNPO55XMKGDL6RNFHFOSTVGRYG4BZLF5Z4HNHXKKWLR2X4CHXQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.728735372 (496h42m45.599818417s) [seg_pos={0 0}, upload=435966]: Segment reached optimal threshold of 13 pieces. Dec 26 08:11:45.729023932 (496h42m45.600106988s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:36580->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:11:45.729034102 (496h42m45.600117147s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729235996 (496h42m45.600319061s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 08:11:45.729246326 (496h42m45.600329371s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729268638 (496h42m45.600351673s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 08:11:45.729279359 (496h42m45.600362394s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729311299 (496h42m45.600394344s) [seg_pos={0 0}, upload=435966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:11:45.729417842 (496h42m45.600500887s) [upload=435966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:11:45.729425286 (496h42m45.600508331s) [upload=435966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:11:46.015994742 (496h42m45.887077777s) [upload=435966]: Done uploading segments... Dec 26 08:11:46.016026893 (496h42m45.887109928s) [upload=435966]: Upload finished. err: <nil> Duration: 30.003350151s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.362775257s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:11:40 UTC) orbiter warning: context deadline exceeded Duration: 3.362802228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.345053294s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:11:37 UTC) orbiter warning: context deadline exceeded Duration: 6.345090534s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XX....X...X.....X.....XXX.X......................X..X.XX.] duration: 6.060213141s trace: https://jaeger.dev.storj.io/trace/694c34495d4a1a9c FAIL (2025-12-26 08:10:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:09:27.995905500 (496h40m27.866988545s) [upload=435932]: Starting upload Dec 26 08:09:27.995957480 (496h40m27.867040515s) [upload=435932]: Done starting upload Dec 26 08:09:27.996076125 (496h40m27.86715917s) [upload=435932]: Uploading segments... Dec 26 08:09:27.996164113 (496h40m27.867247158s) [upload=435932]: Got next segment. Inline: false Dec 26 08:09:27.996195343 (496h40m27.867278378s) [upload=435932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:09:27.996200783 (496h40m27.867283818s) [upload=435932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:09:28.245767360 (496h40m28.116850395s) [seg_pos={0 0}, upload=435932]: Begin upload segment... Dec 26 08:09:28.245852092 (496h40m28.116935127s) [seg_pos={0 0}, upload=435932]: Done begin upload segment. Dec 26 08:09:28.245893761 (496h40m28.116976796s) [noise=true , address=145.79.227.16:4911 , piece_id=6OROM3ANMBCJSOLREE7DH52JTXINIGTWPKQTYPMUWTQTDD6EW5DA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.245899321 (496h40m28.116982356s) [noise=true , address=145.79.227.41:4085 , piece_id=TUYVCK7YILQ3EXZSOQNOM3CBK3B7MSVUQU7377GAW6QS7VPXEPKA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.246772374 (496h40m28.117855409s) [noise=true , address=77.90.31.14:4664 , piece_id=L62VLUSGXFQOH7V5EXJH7TD67QFMAVDK632UIY7ATLSJS53X35FA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247111419 (496h40m28.118194454s) [noise=true , address=145.79.227.3:4149 , piece_id=EC3QBAHCXPQ7SSSQYBVM3S5KW4SSRGZUF3WHWMSR2U36DSBLWITA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247467898 (496h40m28.118550943s) [noise=true , address=45.91.116.11:4118 , piece_id=27IMZC6QT6YAOPH6FZW5YISPRPWILTNFTPEEEFSRXC6ETBL6HNNA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247903156 (496h40m28.118986191s) [noise=true , address=45.91.116.12:4228 , piece_id=NUJ7H6RDKL53Q3BOCDORFE2HJPFQC6AYZXGMIS5FKE3QJ7ODM5QA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.248341361 (496h40m28.119424396s) [noise=true , address=45.91.116.8:4047 , piece_id=BXCX32GYWGBWTEHO6RFD7OW7GVV44JXIIBHGCOU6J242WNFSEMSQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.248762924 (496h40m28.119845959s) [noise=true , address=45.91.116.9:4261 , piece_id=XIY37R3GGNYLVBFWMCKZFBCA732IMJS4OJXET74Y72IQMYOGGUUA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.249157806 (496h40m28.120240841s) [noise=true , address=45.91.116.14:4190 , piece_id=WAWW2IC7YVD3YT3NLAVM432M2QFOWG3KF6ULPRM2Q4V7EKVYYWSA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.250531121 (496h40m28.121614166s) [noise=true , address=145.79.227.46:4603 , piece_id=CVDBVWOQ4SY5QOXWEPCPGQM5CNFDTZVD67DBUSSEBKGXFPDKDNMQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.250948365 (496h40m28.1220314s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.251809304 (496h40m28.122892339s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.253211865 (496h40m28.124294911s) [noise=true , address=45.91.116.5:4297 , piece_id=DY6U7X577KRNKA6GQWY3OCXYDQFY2ZBDWH7V3K5LMIOQ5TSWINEQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.253946885 (496h40m28.12502992s) [noise=true , address=145.79.227.19:4078 , piece_id=C5WY2NLSKRM3IYZWISLCNDUW6Y33L7IH6DVVX4ZZP2GIGO7MRNMA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.258262212 (496h40m28.129345257s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.269630158 (496h40m28.140713193s) [noise=true , address=145.79.227.51:4410 , piece_id=J6FQSXKS5DWDLB5OQMYSVCKLEQOMOL73HVTXXXTL4ARBZQILVCPQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:29.978129477 (496h40m29.849212522s) [upload=435932]: Next returned nil segment Dec 26 08:09:29.978145057 (496h40m29.849228102s) [upload=435932]: Waiting for error group managing segments... Dec 26 08:09:30.266733918 (496h40m30.137816953s) [noise=true , address=145.79.227.41:4085 , piece_id=TUYVCK7YILQ3EXZSOQNOM3CBK3B7MSVUQU7377GAW6QS7VPXEPKA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.267100977 (496h40m30.138184023s) [noise=true , address=145.79.227.16:4911 , piece_id=6OROM3ANMBCJSOLREE7DH52JTXINIGTWPKQTYPMUWTQTDD6EW5DA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.273766499 (496h40m30.144849544s) [noise=true , address=145.79.227.46:4603 , piece_id=CVDBVWOQ4SY5QOXWEPCPGQM5CNFDTZVD67DBUSSEBKGXFPDKDNMQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.279601881 (496h40m30.150684926s) [noise=true , address=145.79.227.51:4410 , piece_id=J6FQSXKS5DWDLB5OQMYSVCKLEQOMOL73HVTXXXTL4ARBZQILVCPQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.285658013 (496h40m30.156741058s) [noise=true , address=145.79.227.19:4078 , piece_id=C5WY2NLSKRM3IYZWISLCNDUW6Y33L7IH6DVVX4ZZP2GIGO7MRNMA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.290868184 (496h40m30.161951239s) [noise=true , address=145.79.227.3:4149 , piece_id=EC3QBAHCXPQ7SSSQYBVM3S5KW4SSRGZUF3WHWMSR2U36DSBLWITA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.502516858 (496h40m30.373599903s) [noise=true , address=45.91.116.11:4118 , piece_id=27IMZC6QT6YAOPH6FZW5YISPRPWILTNFTPEEEFSRXC6ETBL6HNNA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.553673489 (496h40m30.424756524s) [noise=true , address=45.91.116.8:4047 , piece_id=BXCX32GYWGBWTEHO6RFD7OW7GVV44JXIIBHGCOU6J242WNFSEMSQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.606375212 (496h40m30.477458247s) [noise=true , address=45.91.116.9:4261 , piece_id=XIY37R3GGNYLVBFWMCKZFBCA732IMJS4OJXET74Y72IQMYOGGUUA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.608045683 (496h40m30.479128718s) [noise=true , address=45.91.116.14:4190 , piece_id=WAWW2IC7YVD3YT3NLAVM432M2QFOWG3KF6ULPRM2Q4V7EKVYYWSA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.613178417 (496h40m30.484261462s) [noise=true , address=45.91.116.5:4297 , piece_id=DY6U7X577KRNKA6GQWY3OCXYDQFY2ZBDWH7V3K5LMIOQ5TSWINEQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.619262832 (496h40m30.490345867s) [noise=true , address=45.91.116.12:4228 , piece_id=NUJ7H6RDKL53Q3BOCDORFE2HJPFQC6AYZXGMIS5FKE3QJ7ODM5QA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.644609176 (496h40m30.515692211s) [noise=true , address=77.90.31.14:4664 , piece_id=L62VLUSGXFQOH7V5EXJH7TD67QFMAVDK632UIY7ATLSJS53X35FA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.644627411 (496h40m30.515710456s) [seg_pos={0 0}, upload=435932]: Segment reached optimal threshold of 13 pieces. Dec 26 08:09:30.644812703 (496h40m30.515895738s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.644819887 (496h40m30.515902922s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.644944875 (496h40m30.51602791s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.644963070 (496h40m30.516046115s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.645036961 (496h40m30.516119996s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.645045367 (496h40m30.516128402s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.645093338 (496h40m30.516176373s) [seg_pos={0 0}, upload=435932]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:09:30.645284733 (496h40m30.516367768s) [upload=435932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:09:30.645294180 (496h40m30.516377225s) [upload=435932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:09:30.938761932 (496h40m30.809844967s) [upload=435932]: Done uploading segments... Dec 26 08:09:30.938782241 (496h40m30.809865276s) [upload=435932]: Upload finished. err: <nil> Duration: 6.060239932s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.596541877s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:10:36 UTC) orbiter warning: context deadline exceeded Duration: 3.596569449s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.902341998s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:10:32 UTC) orbiter warning: context deadline exceeded Duration: 5.90236948s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....XX....X...X.....X.....XXX.X......................X..X.XX] duration: 30.002686777s trace: https://jaeger.dev.storj.io/trace/694c34495d4a1a9c FAIL (2025-12-26 08:09:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:09:27.995905500 (496h40m27.866988545s) [upload=435932]: Starting upload Dec 26 08:09:27.995957480 (496h40m27.867040515s) [upload=435932]: Done starting upload Dec 26 08:09:27.996076125 (496h40m27.86715917s) [upload=435932]: Uploading segments... Dec 26 08:09:27.996164113 (496h40m27.867247158s) [upload=435932]: Got next segment. Inline: false Dec 26 08:09:27.996195343 (496h40m27.867278378s) [upload=435932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:09:27.996200783 (496h40m27.867283818s) [upload=435932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:09:28.245767360 (496h40m28.116850395s) [seg_pos={0 0}, upload=435932]: Begin upload segment... Dec 26 08:09:28.245852092 (496h40m28.116935127s) [seg_pos={0 0}, upload=435932]: Done begin upload segment. Dec 26 08:09:28.245893761 (496h40m28.116976796s) [noise=true , address=145.79.227.16:4911 , piece_id=6OROM3ANMBCJSOLREE7DH52JTXINIGTWPKQTYPMUWTQTDD6EW5DA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.245899321 (496h40m28.116982356s) [noise=true , address=145.79.227.41:4085 , piece_id=TUYVCK7YILQ3EXZSOQNOM3CBK3B7MSVUQU7377GAW6QS7VPXEPKA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.246772374 (496h40m28.117855409s) [noise=true , address=77.90.31.14:4664 , piece_id=L62VLUSGXFQOH7V5EXJH7TD67QFMAVDK632UIY7ATLSJS53X35FA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247111419 (496h40m28.118194454s) [noise=true , address=145.79.227.3:4149 , piece_id=EC3QBAHCXPQ7SSSQYBVM3S5KW4SSRGZUF3WHWMSR2U36DSBLWITA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247467898 (496h40m28.118550943s) [noise=true , address=45.91.116.11:4118 , piece_id=27IMZC6QT6YAOPH6FZW5YISPRPWILTNFTPEEEFSRXC6ETBL6HNNA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247903156 (496h40m28.118986191s) [noise=true , address=45.91.116.12:4228 , piece_id=NUJ7H6RDKL53Q3BOCDORFE2HJPFQC6AYZXGMIS5FKE3QJ7ODM5QA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.248341361 (496h40m28.119424396s) [noise=true , address=45.91.116.8:4047 , piece_id=BXCX32GYWGBWTEHO6RFD7OW7GVV44JXIIBHGCOU6J242WNFSEMSQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.248762924 (496h40m28.119845959s) [noise=true , address=45.91.116.9:4261 , piece_id=XIY37R3GGNYLVBFWMCKZFBCA732IMJS4OJXET74Y72IQMYOGGUUA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.249157806 (496h40m28.120240841s) [noise=true , address=45.91.116.14:4190 , piece_id=WAWW2IC7YVD3YT3NLAVM432M2QFOWG3KF6ULPRM2Q4V7EKVYYWSA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.250531121 (496h40m28.121614166s) [noise=true , address=145.79.227.46:4603 , piece_id=CVDBVWOQ4SY5QOXWEPCPGQM5CNFDTZVD67DBUSSEBKGXFPDKDNMQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.250948365 (496h40m28.1220314s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.251809304 (496h40m28.122892339s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.253211865 (496h40m28.124294911s) [noise=true , address=45.91.116.5:4297 , piece_id=DY6U7X577KRNKA6GQWY3OCXYDQFY2ZBDWH7V3K5LMIOQ5TSWINEQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.253946885 (496h40m28.12502992s) [noise=true , address=145.79.227.19:4078 , piece_id=C5WY2NLSKRM3IYZWISLCNDUW6Y33L7IH6DVVX4ZZP2GIGO7MRNMA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.258262212 (496h40m28.129345257s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.269630158 (496h40m28.140713193s) [noise=true , address=145.79.227.51:4410 , piece_id=J6FQSXKS5DWDLB5OQMYSVCKLEQOMOL73HVTXXXTL4ARBZQILVCPQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:29.978129477 (496h40m29.849212522s) [upload=435932]: Next returned nil segment Dec 26 08:09:29.978145057 (496h40m29.849228102s) [upload=435932]: Waiting for error group managing segments... Dec 26 08:09:30.266733918 (496h40m30.137816953s) [noise=true , address=145.79.227.41:4085 , piece_id=TUYVCK7YILQ3EXZSOQNOM3CBK3B7MSVUQU7377GAW6QS7VPXEPKA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.267100977 (496h40m30.138184023s) [noise=true , address=145.79.227.16:4911 , piece_id=6OROM3ANMBCJSOLREE7DH52JTXINIGTWPKQTYPMUWTQTDD6EW5DA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.273766499 (496h40m30.144849544s) [noise=true , address=145.79.227.46:4603 , piece_id=CVDBVWOQ4SY5QOXWEPCPGQM5CNFDTZVD67DBUSSEBKGXFPDKDNMQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.279601881 (496h40m30.150684926s) [noise=true , address=145.79.227.51:4410 , piece_id=J6FQSXKS5DWDLB5OQMYSVCKLEQOMOL73HVTXXXTL4ARBZQILVCPQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.285658013 (496h40m30.156741058s) [noise=true , address=145.79.227.19:4078 , piece_id=C5WY2NLSKRM3IYZWISLCNDUW6Y33L7IH6DVVX4ZZP2GIGO7MRNMA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.290868184 (496h40m30.161951239s) [noise=true , address=145.79.227.3:4149 , piece_id=EC3QBAHCXPQ7SSSQYBVM3S5KW4SSRGZUF3WHWMSR2U36DSBLWITA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.502516858 (496h40m30.373599903s) [noise=true , address=45.91.116.11:4118 , piece_id=27IMZC6QT6YAOPH6FZW5YISPRPWILTNFTPEEEFSRXC6ETBL6HNNA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.553673489 (496h40m30.424756524s) [noise=true , address=45.91.116.8:4047 , piece_id=BXCX32GYWGBWTEHO6RFD7OW7GVV44JXIIBHGCOU6J242WNFSEMSQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.606375212 (496h40m30.477458247s) [noise=true , address=45.91.116.9:4261 , piece_id=XIY37R3GGNYLVBFWMCKZFBCA732IMJS4OJXET74Y72IQMYOGGUUA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.608045683 (496h40m30.479128718s) [noise=true , address=45.91.116.14:4190 , piece_id=WAWW2IC7YVD3YT3NLAVM432M2QFOWG3KF6ULPRM2Q4V7EKVYYWSA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.613178417 (496h40m30.484261462s) [noise=true , address=45.91.116.5:4297 , piece_id=DY6U7X577KRNKA6GQWY3OCXYDQFY2ZBDWH7V3K5LMIOQ5TSWINEQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.619262832 (496h40m30.490345867s) [noise=true , address=45.91.116.12:4228 , piece_id=NUJ7H6RDKL53Q3BOCDORFE2HJPFQC6AYZXGMIS5FKE3QJ7ODM5QA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.644609176 (496h40m30.515692211s) [noise=true , address=77.90.31.14:4664 , piece_id=L62VLUSGXFQOH7V5EXJH7TD67QFMAVDK632UIY7ATLSJS53X35FA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.644627411 (496h40m30.515710456s) [seg_pos={0 0}, upload=435932]: Segment reached optimal threshold of 13 pieces. Dec 26 08:09:30.644812703 (496h40m30.515895738s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.644819887 (496h40m30.515902922s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.644944875 (496h40m30.51602791s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.644963070 (496h40m30.516046115s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.645036961 (496h40m30.516119996s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.645045367 (496h40m30.516128402s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.645093338 (496h40m30.516176373s) [seg_pos={0 0}, upload=435932]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:09:30.645284733 (496h40m30.516367768s) [upload=435932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:09:30.645294180 (496h40m30.516377225s) [upload=435932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:09:30.938761932 (496h40m30.809844967s) [upload=435932]: Done uploading segments... Dec 26 08:09:30.938782241 (496h40m30.809865276s) [upload=435932]: Upload finished. err: <nil> Duration: 30.002695293s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.210570034s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:09:44 UTC) orbiter warning: context deadline exceeded Duration: 6.210658493s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.71583842s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:09:39 UTC) orbiter warning: context deadline exceeded Duration: 3.715866122s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001188976s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:09:35 UTC) orbiter warning: context deadline exceeded Duration: 30.001213212s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....XX....X...X.....X.....XXX.X......................X..X.X] duration: 30.003382302s trace: https://jaeger.dev.storj.io/trace/2cc3cf45f0b4f40a FAIL (2025-12-26 08:09:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CXFZLHRRVMWK45GFJM7PDG7G4GWF5XOIBCIDHEAQWL2VX2WPIUDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: AKOSKSLPETP6L24NFDADZJJZYJBVJ5CA2JRJI4XTE6RPXGQ2VHBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: V7Y6RCNZ4LJAA2V76KDEFYFX3DTZYNM4B6LRZDZ4CAWCFDHHLCFQ) context deadline exceeded ======LOGS====== Dec 26 08:08:57.775505721 (496h39m57.646588756s) [upload=435927]: Starting upload Dec 26 08:08:57.775578029 (496h39m57.646661064s) [upload=435927]: Done starting upload Dec 26 08:08:57.775821673 (496h39m57.646904718s) [upload=435927]: Uploading segments... Dec 26 08:08:57.775975425 (496h39m57.64705847s) [upload=435927]: Got next segment. Inline: false Dec 26 08:08:57.776761893 (496h39m57.647844938s) [upload=435927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:08:57.776791500 (496h39m57.647874545s) [upload=435927]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:08:58.050026344 (496h39m57.921109389s) [seg_pos={0 0}, upload=435927]: Begin upload segment... Dec 26 08:08:58.050092410 (496h39m57.921175455s) [seg_pos={0 0}, upload=435927]: Done begin upload segment. Dec 26 08:08:58.050162283 (496h39m57.921245318s) [noise=true , address=145.79.227.42:4123 , piece_id=IIAMXKL2WYPNZB2PHZEMVCLC7XIVTNJSPAEBMJCGLV2NUUWPIWBQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.050174325 (496h39m57.92125737s) [noise=true , address=45.91.116.14:4192 , piece_id=V7Y6RCNZ4LJAA2V76KDEFYFX3DTZYNM4B6LRZDZ4CAWCFDHHLCFQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.051811232 (496h39m57.922894267s) [noise=true , address=145.79.227.38:4979 , piece_id=ISF2OUO6QI5GUW7SAHNIG7SHHCMELHIIXZIS24HJAM6QR3J4NG4Q, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.051953854 (496h39m57.923036899s) [noise=true , address=77.90.31.10:4519 , piece_id=FBXQL7VBVLIY4S7DVMG4A26PW6SI6NDLBWGBJOVZXOVKZHHNOJGQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.052331854 (496h39m57.923414889s) [noise=true , address=45.91.116.8:4046 , piece_id=CXFZLHRRVMWK45GFJM7PDG7G4GWF5XOIBCIDHEAQWL2VX2WPIUDA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.052438937 (496h39m57.923521982s) [noise=true , address=77.90.31.19:4844 , piece_id=RNH7QMZTAGKJS7OQJ6KEOKM2AFAULB2T24UFPOGO5VDU3GEH5K7A, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.052726735 (496h39m57.92380977s) [noise=true , address=145.79.227.48:4301 , piece_id=6FRHKBQQGUOIRP6VAYWNKZKRT7BJJUQCKHOHSZBFNXFQNZM3346Q, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.052933098 (496h39m57.924016143s) [noise=true , address=145.79.227.45:4230 , piece_id=T2TST2XNNHPXOKQPFVX6UYTMJX7CB2GHSWKZ6UKKQDTY5FBN7GTA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.053113142 (496h39m57.924196187s) [noise=true , address=45.91.116.10:4084 , piece_id=ANRH3APVGCFKORPVBPBD22RHDYBYUZLIMNBAXMIUF7SHJ2PVNBFQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.053419104 (496h39m57.924502139s) [noise=true , address=77.90.31.16:4736 , piece_id=AKOSKSLPETP6L24NFDADZJJZYJBVJ5CA2JRJI4XTE6RPXGQ2VHBQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.054406274 (496h39m57.925489309s) [noise=true , address=145.79.227.19:4078 , piece_id=QMU5C6DJATPUTHIRH47W3KCXFOWGSBVWY2G7VV45ALZPQEJKYZSQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.054990486 (496h39m57.926073521s) [noise=true , address=145.79.227.32:4835 , piece_id=YSQA4TXGWOOSRIIMSHBRP25JOAXGLMYOTW3JHACKKCSGICTHBXOA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.055052985 (496h39m57.92613602s) [noise=true , address=77.90.31.21:4880 , piece_id=6AUYDALDN32IVYYSE2D72HGULS6TYY5YRZ3ISAYOZ4JMARIQUQFQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.055610176 (496h39m57.926693231s) [noise=true , address=145.79.227.5:4221 , piece_id=5L3INSDWHHGZHTWJ6IQVQKHXGP47PICZGUGH23V43PGX2H6JFHDA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.056751559 (496h39m57.927834605s) [noise=true , address=145.79.227.18:4007 , piece_id=LFXWJ3TTUVC5KZVJV4BUNJ2ML2IKCURJMMFNCY3ZR6PKQIPXDGPA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.057311806 (496h39m57.928394841s) [noise=true , address=45.91.116.9:4263 , piece_id=MWFHF3TEJGENCED5Z5LMSJE5DFPOJSRLC6MWVOZZS6QYEUWZ7HOQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:59.765097205 (496h39m59.63618024s) [upload=435927]: Next returned nil segment Dec 26 08:08:59.765116653 (496h39m59.636199688s) [upload=435927]: Waiting for error group managing segments... Dec 26 08:09:00.050879524 (496h39m59.921962559s) [noise=true , address=145.79.227.18:4007 , piece_id=LFXWJ3TTUVC5KZVJV4BUNJ2ML2IKCURJMMFNCY3ZR6PKQIPXDGPA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.054702302 (496h39m59.925785337s) [noise=true , address=145.79.227.19:4078 , piece_id=QMU5C6DJATPUTHIRH47W3KCXFOWGSBVWY2G7VV45ALZPQEJKYZSQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.057171624 (496h39m59.928254669s) [noise=true , address=145.79.227.32:4835 , piece_id=YSQA4TXGWOOSRIIMSHBRP25JOAXGLMYOTW3JHACKKCSGICTHBXOA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.073574328 (496h39m59.944657363s) [noise=true , address=145.79.227.48:4301 , piece_id=6FRHKBQQGUOIRP6VAYWNKZKRT7BJJUQCKHOHSZBFNXFQNZM3346Q, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.074979714 (496h39m59.946062759s) [noise=true , address=145.79.227.38:4979 , piece_id=ISF2OUO6QI5GUW7SAHNIG7SHHCMELHIIXZIS24HJAM6QR3J4NG4Q, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.083596281 (496h39m59.954679316s) [noise=true , address=145.79.227.5:4221 , piece_id=5L3INSDWHHGZHTWJ6IQVQKHXGP47PICZGUGH23V43PGX2H6JFHDA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.084605803 (496h39m59.955688848s) [noise=true , address=145.79.227.42:4123 , piece_id=IIAMXKL2WYPNZB2PHZEMVCLC7XIVTNJSPAEBMJCGLV2NUUWPIWBQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.090001598 (496h39m59.961084653s) [noise=true , address=145.79.227.45:4230 , piece_id=T2TST2XNNHPXOKQPFVX6UYTMJX7CB2GHSWKZ6UKKQDTY5FBN7GTA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.306896503 (496h40m0.177979568s) [noise=true , address=45.91.116.14:4192 , piece_id=V7Y6RCNZ4LJAA2V76KDEFYFX3DTZYNM4B6LRZDZ4CAWCFDHHLCFQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.320347926 (496h40m0.191430961s) [noise=true , address=45.91.116.8:4046 , piece_id=CXFZLHRRVMWK45GFJM7PDG7G4GWF5XOIBCIDHEAQWL2VX2WPIUDA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.369413926 (496h40m0.240496961s) [noise=true , address=77.90.31.16:4736 , piece_id=AKOSKSLPETP6L24NFDADZJJZYJBVJ5CA2JRJI4XTE6RPXGQ2VHBQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.411558997 (496h40m0.282642042s) [noise=true , address=45.91.116.10:4084 , piece_id=ANRH3APVGCFKORPVBPBD22RHDYBYUZLIMNBAXMIUF7SHJ2PVNBFQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.929994685 (496h40m0.801077731s) [noise=true , address=77.90.31.10:4519 , piece_id=FBXQL7VBVLIY4S7DVMG4A26PW6SI6NDLBWGBJOVZXOVKZHHNOJGQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.930018842 (496h40m0.801101887s) [seg_pos={0 0}, upload=435927]: Segment reached optimal threshold of 13 pieces. Dec 26 08:09:00.930554020 (496h40m0.801637066s) [noise=true , address=77.90.31.21:4880 , piece_id=6AUYDALDN32IVYYSE2D72HGULS6TYY5YRZ3ISAYOZ4JMARIQUQFQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:00.930566916 (496h40m0.801649951s) [noise=true , address=77.90.31.21:4880 , piece_id=6AUYDALDN32IVYYSE2D72HGULS6TYY5YRZ3ISAYOZ4JMARIQUQFQ, seg_pos={0 0}, upload=435927]: Long tail cancellation due to optimal threshold Dec 26 08:09:00.930878820 (496h40m0.801961865s) [noise=true , address=45.91.116.9:4263 , piece_id=MWFHF3TEJGENCED5Z5LMSJE5DFPOJSRLC6MWVOZZS6QYEUWZ7HOQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:00.930890712 (496h40m0.801973757s) [noise=true , address=45.91.116.9:4263 , piece_id=MWFHF3TEJGENCED5Z5LMSJE5DFPOJSRLC6MWVOZZS6QYEUWZ7HOQ, seg_pos={0 0}, upload=435927]: Long tail cancellation due to optimal threshold Dec 26 08:09:00.930966696 (496h40m0.802049741s) [noise=true , address=77.90.31.19:4844 , piece_id=RNH7QMZTAGKJS7OQJ6KEOKM2AFAULB2T24UFPOGO5VDU3GEH5K7A, seg_pos={0 0}, upload=435927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: read tcp 188.166.225.170:34724->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:09:00.930992526 (496h40m0.802075571s) [noise=true , address=77.90.31.19:4844 , piece_id=RNH7QMZTAGKJS7OQJ6KEOKM2AFAULB2T24UFPOGO5VDU3GEH5K7A, seg_pos={0 0}, upload=435927]: Long tail cancellation due to optimal threshold Dec 26 08:09:00.931018926 (496h40m0.802101961s) [seg_pos={0 0}, upload=435927]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:09:00.931162299 (496h40m0.802245344s) [upload=435927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:09:00.931170925 (496h40m0.80225397s) [upload=435927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:09:01.226100892 (496h40m1.097183947s) [upload=435927]: Done uploading segments... Dec 26 08:09:01.226158521 (496h40m1.097241566s) [upload=435927]: Upload finished. err: <nil> Duration: 30.003403192s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001419489s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:09:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001442734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....XX....X...X.....X.....XXX.X......................X..X.] duration: 6.115222077s trace: https://jaeger.dev.storj.io/trace/30af03556a5513c5 FAIL (2025-12-26 08:07:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ) context deadline exceeded ======LOGS====== Dec 26 08:07:07.906767540 (496h38m7.777850585s) [upload=435902]: Starting upload Dec 26 08:07:07.906847712 (496h38m7.777930757s) [upload=435902]: Done starting upload Dec 26 08:07:07.906950288 (496h38m7.778033333s) [upload=435902]: Uploading segments... Dec 26 08:07:07.907089693 (496h38m7.778172738s) [upload=435902]: Got next segment. Inline: false Dec 26 08:07:07.907123157 (496h38m7.778206202s) [upload=435902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:07:07.907128818 (496h38m7.778211863s) [upload=435902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:07:08.156507276 (496h38m8.027590321s) [seg_pos={0 0}, upload=435902]: Begin upload segment... Dec 26 08:07:08.156552783 (496h38m8.027635818s) [seg_pos={0 0}, upload=435902]: Done begin upload segment. Dec 26 08:07:08.156588841 (496h38m8.027671886s) [noise=true , address=145.79.227.4:4185 , piece_id=FBPU4HO6MRVSXGZZ6RU5N76Z7CUEZGGHW22FSDMZYWDGVHTGFSMQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.156650349 (496h38m8.027733394s) [noise=true , address=77.90.31.16:4736 , piece_id=CHQHMCKJ6RLQO43OWJNIV2AFDZXDULKPBODP3IB7NCHQU2O5R45Q, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157120024 (496h38m8.028203069s) [noise=true , address=45.91.116.13:4155 , piece_id=UDRMOUCQDALXRIEYHGQXU5G7QZPFSDLCFIVITD7WMUBFOXZAI6KQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157494075 (496h38m8.02857711s) [noise=true , address=145.79.227.3:4150 , piece_id=OPEEED2VTFN3DGOMOC6PRAV3VC6XQTVLYGML7CXK26XQ5KAEV5MA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157949754 (496h38m8.029032789s) [noise=true , address=145.79.227.46:4603 , piece_id=JSL3WMUJCBNM6U3F3YXC23O42NEYWUYT6I5LFOFT6LCP6RZ6IBMA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.158274692 (496h38m8.029357737s) [noise=true , address=145.79.227.11:4730 , piece_id=UX4BQ7EVZXIDMVX7PTUH67W3TXHJO52RR2KUF7VSRL2KMYAPVHSQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.160792296 (496h38m8.031875331s) [noise=true , address=145.79.227.51:4409 , piece_id=4HRMOJJKF3K3ONPWY6YC7V3LOZ2MJMJBK2G36HULCMHJEUXLH2AQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.162033850 (496h38m8.033116895s) [noise=true , address=145.79.227.17:4970 , piece_id=P3QDN2Z2LPS66PVL55JF4PEYHI3WYHNGVOFLV5EUUVDEP6TROURA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.163141009 (496h38m8.034224054s) [noise=true , address=45.91.116.10:4084 , piece_id=S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.163484513 (496h38m8.034567558s) [noise=true , address=145.79.227.49:4339 , piece_id=IOYNBKYW3DHULYJDVMNH34F4GY7GRU7BOKO2FO4HPDLGESPHIYHA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.164352194 (496h38m8.035435239s) [noise=true , address=45.91.116.12:4227 , piece_id=QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.165066104 (496h38m8.036149139s) [noise=true , address=77.90.31.15:4697 , piece_id=AENGNHGGOJFQHE6KMQEZYMC6BEZCRNEGKOZG2G7VA3KYUSG3SBCQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.165484070 (496h38m8.036567105s) [noise=true , address=77.90.31.12:4589 , piece_id=N7CZHZEIBLAGZKKXKPXERI2QIKIR424ZHJMJKVA36UF3SNLBL4FQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.167330015 (496h38m8.03841306s) [noise=true , address=77.90.31.18:4805 , piece_id=KWP3F453PXGCWYGQBGJCMW4IIMYUUGWRNHYJRLN2VNWGDPQHCV2A, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.168621784 (496h38m8.03970482s) [noise=true , address=145.79.227.19:4078 , piece_id=TLFK7MBN5OJERNXCXFDSVUIQUFF4A6654U52UK2RNS4Z2D7K2YLQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.169725707 (496h38m8.040808742s) [noise=true , address=145.79.227.47:4267 , piece_id=V37QLFJTKR72T7BUELFPHQJOIDK4NN44NPDVH3IKXDPLBQPR3VKQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:09.349050539 (496h38m9.220133594s) [noise=true , address=77.90.31.12:4589 , piece_id=N7CZHZEIBLAGZKKXKPXERI2QIKIR424ZHJMJKVA36UF3SNLBL4FQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:09.622296024 (496h38m9.493379079s) [noise=true , address=77.90.31.18:4805 , piece_id=KWP3F453PXGCWYGQBGJCMW4IIMYUUGWRNHYJRLN2VNWGDPQHCV2A, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:09.893726172 (496h38m9.764809207s) [upload=435902]: Next returned nil segment Dec 26 08:07:09.893739698 (496h38m9.764822743s) [upload=435902]: Waiting for error group managing segments... Dec 26 08:07:10.176986808 (496h38m10.048069853s) [noise=true , address=145.79.227.51:4409 , piece_id=4HRMOJJKF3K3ONPWY6YC7V3LOZ2MJMJBK2G36HULCMHJEUXLH2AQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.184413820 (496h38m10.055496865s) [noise=true , address=145.79.227.3:4150 , piece_id=OPEEED2VTFN3DGOMOC6PRAV3VC6XQTVLYGML7CXK26XQ5KAEV5MA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.189827909 (496h38m10.060910944s) [noise=true , address=145.79.227.19:4078 , piece_id=TLFK7MBN5OJERNXCXFDSVUIQUFF4A6654U52UK2RNS4Z2D7K2YLQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.190219355 (496h38m10.0613024s) [noise=true , address=145.79.227.17:4970 , piece_id=P3QDN2Z2LPS66PVL55JF4PEYHI3WYHNGVOFLV5EUUVDEP6TROURA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.202474541 (496h38m10.073557576s) [noise=true , address=145.79.227.11:4730 , piece_id=UX4BQ7EVZXIDMVX7PTUH67W3TXHJO52RR2KUF7VSRL2KMYAPVHSQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.203448456 (496h38m10.074531491s) [noise=true , address=145.79.227.4:4185 , piece_id=FBPU4HO6MRVSXGZZ6RU5N76Z7CUEZGGHW22FSDMZYWDGVHTGFSMQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.206554119 (496h38m10.077637154s) [noise=true , address=145.79.227.46:4603 , piece_id=JSL3WMUJCBNM6U3F3YXC23O42NEYWUYT6I5LFOFT6LCP6RZ6IBMA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.210487950 (496h38m10.081570985s) [noise=true , address=145.79.227.49:4339 , piece_id=IOYNBKYW3DHULYJDVMNH34F4GY7GRU7BOKO2FO4HPDLGESPHIYHA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.229236820 (496h38m10.100319855s) [noise=true , address=145.79.227.47:4267 , piece_id=V37QLFJTKR72T7BUELFPHQJOIDK4NN44NPDVH3IKXDPLBQPR3VKQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.416501755 (496h38m10.2875848s) [noise=true , address=45.91.116.13:4155 , piece_id=UDRMOUCQDALXRIEYHGQXU5G7QZPFSDLCFIVITD7WMUBFOXZAI6KQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.724923190 (496h38m10.596006235s) [noise=true , address=45.91.116.12:4227 , piece_id=QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:12.179810177 (496h38m12.050893222s) [noise=true , address=45.91.116.10:4084 , piece_id=S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:12.480684464 (496h38m12.351767499s) [noise=true , address=77.90.31.15:4697 , piece_id=AENGNHGGOJFQHE6KMQEZYMC6BEZCRNEGKOZG2G7VA3KYUSG3SBCQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:13.352037555 (496h38m13.2231206s) [noise=true , address=77.90.31.16:4736 , piece_id=CHQHMCKJ6RLQO43OWJNIV2AFDZXDULKPBODP3IB7NCHQU2O5R45Q, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:13.352077571 (496h38m13.223160616s) [seg_pos={0 0}, upload=435902]: Segment reached optimal threshold of 13 pieces. Dec 26 08:07:13.352273996 (496h38m13.223357041s) [seg_pos={0 0}, upload=435902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:07:13.352363626 (496h38m13.223446671s) [upload=435902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:07:13.352370519 (496h38m13.223453564s) [upload=435902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:07:13.646491094 (496h38m13.517574129s) [upload=435902]: Done uploading segments... Dec 26 08:07:13.646512355 (496h38m13.51759539s) [upload=435902]: Upload finished. err: <nil> Duration: 6.115239611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.....XX....X...X.....X.....XXX.X......................X..X] duration: 30.004076735s trace: https://jaeger.dev.storj.io/trace/30af03556a5513c5 FAIL (2025-12-26 08:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ) context deadline exceeded ======LOGS====== Dec 26 08:07:07.906767540 (496h38m7.777850585s) [upload=435902]: Starting upload Dec 26 08:07:07.906847712 (496h38m7.777930757s) [upload=435902]: Done starting upload Dec 26 08:07:07.906950288 (496h38m7.778033333s) [upload=435902]: Uploading segments... Dec 26 08:07:07.907089693 (496h38m7.778172738s) [upload=435902]: Got next segment. Inline: false Dec 26 08:07:07.907123157 (496h38m7.778206202s) [upload=435902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:07:07.907128818 (496h38m7.778211863s) [upload=435902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:07:08.156507276 (496h38m8.027590321s) [seg_pos={0 0}, upload=435902]: Begin upload segment... Dec 26 08:07:08.156552783 (496h38m8.027635818s) [seg_pos={0 0}, upload=435902]: Done begin upload segment. Dec 26 08:07:08.156588841 (496h38m8.027671886s) [noise=true , address=145.79.227.4:4185 , piece_id=FBPU4HO6MRVSXGZZ6RU5N76Z7CUEZGGHW22FSDMZYWDGVHTGFSMQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.156650349 (496h38m8.027733394s) [noise=true , address=77.90.31.16:4736 , piece_id=CHQHMCKJ6RLQO43OWJNIV2AFDZXDULKPBODP3IB7NCHQU2O5R45Q, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157120024 (496h38m8.028203069s) [noise=true , address=45.91.116.13:4155 , piece_id=UDRMOUCQDALXRIEYHGQXU5G7QZPFSDLCFIVITD7WMUBFOXZAI6KQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157494075 (496h38m8.02857711s) [noise=true , address=145.79.227.3:4150 , piece_id=OPEEED2VTFN3DGOMOC6PRAV3VC6XQTVLYGML7CXK26XQ5KAEV5MA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157949754 (496h38m8.029032789s) [noise=true , address=145.79.227.46:4603 , piece_id=JSL3WMUJCBNM6U3F3YXC23O42NEYWUYT6I5LFOFT6LCP6RZ6IBMA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.158274692 (496h38m8.029357737s) [noise=true , address=145.79.227.11:4730 , piece_id=UX4BQ7EVZXIDMVX7PTUH67W3TXHJO52RR2KUF7VSRL2KMYAPVHSQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.160792296 (496h38m8.031875331s) [noise=true , address=145.79.227.51:4409 , piece_id=4HRMOJJKF3K3ONPWY6YC7V3LOZ2MJMJBK2G36HULCMHJEUXLH2AQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.162033850 (496h38m8.033116895s) [noise=true , address=145.79.227.17:4970 , piece_id=P3QDN2Z2LPS66PVL55JF4PEYHI3WYHNGVOFLV5EUUVDEP6TROURA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.163141009 (496h38m8.034224054s) [noise=true , address=45.91.116.10:4084 , piece_id=S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.163484513 (496h38m8.034567558s) [noise=true , address=145.79.227.49:4339 , piece_id=IOYNBKYW3DHULYJDVMNH34F4GY7GRU7BOKO2FO4HPDLGESPHIYHA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.164352194 (496h38m8.035435239s) [noise=true , address=45.91.116.12:4227 , piece_id=QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.165066104 (496h38m8.036149139s) [noise=true , address=77.90.31.15:4697 , piece_id=AENGNHGGOJFQHE6KMQEZYMC6BEZCRNEGKOZG2G7VA3KYUSG3SBCQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.165484070 (496h38m8.036567105s) [noise=true , address=77.90.31.12:4589 , piece_id=N7CZHZEIBLAGZKKXKPXERI2QIKIR424ZHJMJKVA36UF3SNLBL4FQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.167330015 (496h38m8.03841306s) [noise=true , address=77.90.31.18:4805 , piece_id=KWP3F453PXGCWYGQBGJCMW4IIMYUUGWRNHYJRLN2VNWGDPQHCV2A, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.168621784 (496h38m8.03970482s) [noise=true , address=145.79.227.19:4078 , piece_id=TLFK7MBN5OJERNXCXFDSVUIQUFF4A6654U52UK2RNS4Z2D7K2YLQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.169725707 (496h38m8.040808742s) [noise=true , address=145.79.227.47:4267 , piece_id=V37QLFJTKR72T7BUELFPHQJOIDK4NN44NPDVH3IKXDPLBQPR3VKQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:09.349050539 (496h38m9.220133594s) [noise=true , address=77.90.31.12:4589 , piece_id=N7CZHZEIBLAGZKKXKPXERI2QIKIR424ZHJMJKVA36UF3SNLBL4FQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:09.622296024 (496h38m9.493379079s) [noise=true , address=77.90.31.18:4805 , piece_id=KWP3F453PXGCWYGQBGJCMW4IIMYUUGWRNHYJRLN2VNWGDPQHCV2A, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:09.893726172 (496h38m9.764809207s) [upload=435902]: Next returned nil segment Dec 26 08:07:09.893739698 (496h38m9.764822743s) [upload=435902]: Waiting for error group managing segments... Dec 26 08:07:10.176986808 (496h38m10.048069853s) [noise=true , address=145.79.227.51:4409 , piece_id=4HRMOJJKF3K3ONPWY6YC7V3LOZ2MJMJBK2G36HULCMHJEUXLH2AQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.184413820 (496h38m10.055496865s) [noise=true , address=145.79.227.3:4150 , piece_id=OPEEED2VTFN3DGOMOC6PRAV3VC6XQTVLYGML7CXK26XQ5KAEV5MA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.189827909 (496h38m10.060910944s) [noise=true , address=145.79.227.19:4078 , piece_id=TLFK7MBN5OJERNXCXFDSVUIQUFF4A6654U52UK2RNS4Z2D7K2YLQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.190219355 (496h38m10.0613024s) [noise=true , address=145.79.227.17:4970 , piece_id=P3QDN2Z2LPS66PVL55JF4PEYHI3WYHNGVOFLV5EUUVDEP6TROURA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.202474541 (496h38m10.073557576s) [noise=true , address=145.79.227.11:4730 , piece_id=UX4BQ7EVZXIDMVX7PTUH67W3TXHJO52RR2KUF7VSRL2KMYAPVHSQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.203448456 (496h38m10.074531491s) [noise=true , address=145.79.227.4:4185 , piece_id=FBPU4HO6MRVSXGZZ6RU5N76Z7CUEZGGHW22FSDMZYWDGVHTGFSMQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.206554119 (496h38m10.077637154s) [noise=true , address=145.79.227.46:4603 , piece_id=JSL3WMUJCBNM6U3F3YXC23O42NEYWUYT6I5LFOFT6LCP6RZ6IBMA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.210487950 (496h38m10.081570985s) [noise=true , address=145.79.227.49:4339 , piece_id=IOYNBKYW3DHULYJDVMNH34F4GY7GRU7BOKO2FO4HPDLGESPHIYHA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.229236820 (496h38m10.100319855s) [noise=true , address=145.79.227.47:4267 , piece_id=V37QLFJTKR72T7BUELFPHQJOIDK4NN44NPDVH3IKXDPLBQPR3VKQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.416501755 (496h38m10.2875848s) [noise=true , address=45.91.116.13:4155 , piece_id=UDRMOUCQDALXRIEYHGQXU5G7QZPFSDLCFIVITD7WMUBFOXZAI6KQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.724923190 (496h38m10.596006235s) [noise=true , address=45.91.116.12:4227 , piece_id=QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:12.179810177 (496h38m12.050893222s) [noise=true , address=45.91.116.10:4084 , piece_id=S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:12.480684464 (496h38m12.351767499s) [noise=true , address=77.90.31.15:4697 , piece_id=AENGNHGGOJFQHE6KMQEZYMC6BEZCRNEGKOZG2G7VA3KYUSG3SBCQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:13.352037555 (496h38m13.2231206s) [noise=true , address=77.90.31.16:4736 , piece_id=CHQHMCKJ6RLQO43OWJNIV2AFDZXDULKPBODP3IB7NCHQU2O5R45Q, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:13.352077571 (496h38m13.223160616s) [seg_pos={0 0}, upload=435902]: Segment reached optimal threshold of 13 pieces. Dec 26 08:07:13.352273996 (496h38m13.223357041s) [seg_pos={0 0}, upload=435902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:07:13.352363626 (496h38m13.223446671s) [upload=435902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:07:13.352370519 (496h38m13.223453564s) [upload=435902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:07:13.646491094 (496h38m13.517574129s) [upload=435902]: Done uploading segments... Dec 26 08:07:13.646512355 (496h38m13.51759539s) [upload=435902]: Upload finished. err: <nil> Duration: 30.004106191s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.....XX....X...X.....X.....XXX.X......................X..] duration: 6.074043776s trace: https://jaeger.dev.storj.io/trace/4718c0a3d5f7d0b6 FAIL (2025-12-26 08:06:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : Starting upload Dec 26 08:05:59.308561696 (496h36m59.179644761s) [upload=435884]: Done starting upload Dec 26 08:05:59.308762437 (496h36m59.179845482s) [upload=435884]: Uploading segments... Dec 26 08:05:59.308803346 (496h36m59.179886401s) [upload=435884]: Got next segment. Inline: false Dec 26 08:05:59.308828523 (496h36m59.179911568s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:05:59.308832941 (496h36m59.179915976s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:05:59.554822597 (496h36m59.425905752s) [seg_pos={0 0}, upload=435884]: Begin upload segment... Dec 26 08:05:59.554992110 (496h36m59.426075155s) [seg_pos={0 0}, upload=435884]: Done begin upload segment. Dec 26 08:05:59.555001227 (496h36m59.426084272s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555272563 (496h36m59.426355609s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555831007 (496h36m59.426914052s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.557563516 (496h36m59.428646551s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558176474 (496h36m59.429259529s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558605560 (496h36m59.429688605s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558918105 (496h36m59.430001141s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560223281 (496h36m59.431306326s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560497062 (496h36m59.431580097s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.561239075 (496h36m59.43232211s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562107749 (496h36m59.433190795s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562234630 (496h36m59.433317676s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562980201 (496h36m59.434063246s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564012245 (496h36m59.43509529s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564629551 (496h36m59.435712596s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.565120215 (496h36m59.43620325s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:06:01.176850499 (496h37m1.047933554s) [upload=435884]: Next returned nil segment Dec 26 08:06:01.176869015 (496h37m1.04795206s) [upload=435884]: Waiting for error group managing segments... Dec 26 08:06:01.560358332 (496h37m1.431441387s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.561209874 (496h37m1.432292919s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.567872280 (496h37m1.438955325s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.568944903 (496h37m1.440027948s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.570504894 (496h37m1.441587939s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588111790 (496h37m1.459194836s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588807835 (496h37m1.45989088s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.591812196 (496h37m1.462895241s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.594307557 (496h37m1.465390602s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.597042546 (496h37m1.468125591s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.763606471 (496h37m1.634689516s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.911658847 (496h37m1.782741892s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032411771 (496h37m1.903494826s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032436208 (496h37m1.903519253s) [seg_pos={0 0}, upload=435884]: Segment reached optimal threshold of 13 pieces. Dec 26 08:06:02.032691944 (496h37m1.90377499s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection Dec 26 08:06:02.032701914 (496h37m1.903784959s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032730317 (496h37m1.903813373s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 08:06:02.032740047 (496h37m1.903823102s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032853382 (496h37m1.903936427s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:44234->77.90.31.11:4553: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 08:06:02.032861668 (496h37m1.903944723s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032948333 (496h37m1.904031388s) [seg_pos={0 0}, upload=435884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:06:02.033105061 (496h37m1.904188116s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:06:02.033140599 (496h37m1.904223654s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:06:02.328202557 (496h37m2.199285602s) [upload=435884]: Done uploading segments... Dec 26 08:06:02.328232714 (496h37m2.199315759s) [upload=435884]: Upload finished. err: <nil> Duration: 6.07407761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..X.....XX....X...X.....X.....XXX.X......................X.] duration: 6.046440023s trace: https://jaeger.dev.storj.io/trace/4718c0a3d5f7d0b6 FAIL (2025-12-26 08:06:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : Starting upload Dec 26 08:05:59.308561696 (496h36m59.179644761s) [upload=435884]: Done starting upload Dec 26 08:05:59.308762437 (496h36m59.179845482s) [upload=435884]: Uploading segments... Dec 26 08:05:59.308803346 (496h36m59.179886401s) [upload=435884]: Got next segment. Inline: false Dec 26 08:05:59.308828523 (496h36m59.179911568s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:05:59.308832941 (496h36m59.179915976s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:05:59.554822597 (496h36m59.425905752s) [seg_pos={0 0}, upload=435884]: Begin upload segment... Dec 26 08:05:59.554992110 (496h36m59.426075155s) [seg_pos={0 0}, upload=435884]: Done begin upload segment. Dec 26 08:05:59.555001227 (496h36m59.426084272s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555272563 (496h36m59.426355609s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555831007 (496h36m59.426914052s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.557563516 (496h36m59.428646551s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558176474 (496h36m59.429259529s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558605560 (496h36m59.429688605s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558918105 (496h36m59.430001141s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560223281 (496h36m59.431306326s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560497062 (496h36m59.431580097s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.561239075 (496h36m59.43232211s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562107749 (496h36m59.433190795s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562234630 (496h36m59.433317676s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562980201 (496h36m59.434063246s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564012245 (496h36m59.43509529s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564629551 (496h36m59.435712596s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.565120215 (496h36m59.43620325s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:06:01.176850499 (496h37m1.047933554s) [upload=435884]: Next returned nil segment Dec 26 08:06:01.176869015 (496h37m1.04795206s) [upload=435884]: Waiting for error group managing segments... Dec 26 08:06:01.560358332 (496h37m1.431441387s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.561209874 (496h37m1.432292919s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.567872280 (496h37m1.438955325s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.568944903 (496h37m1.440027948s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.570504894 (496h37m1.441587939s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588111790 (496h37m1.459194836s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588807835 (496h37m1.45989088s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.591812196 (496h37m1.462895241s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.594307557 (496h37m1.465390602s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.597042546 (496h37m1.468125591s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.763606471 (496h37m1.634689516s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.911658847 (496h37m1.782741892s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032411771 (496h37m1.903494826s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032436208 (496h37m1.903519253s) [seg_pos={0 0}, upload=435884]: Segment reached optimal threshold of 13 pieces. Dec 26 08:06:02.032691944 (496h37m1.90377499s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection Dec 26 08:06:02.032701914 (496h37m1.903784959s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032730317 (496h37m1.903813373s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 08:06:02.032740047 (496h37m1.903823102s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032853382 (496h37m1.903936427s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:44234->77.90.31.11:4553: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 08:06:02.032861668 (496h37m1.903944723s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032948333 (496h37m1.904031388s) [seg_pos={0 0}, upload=435884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:06:02.033105061 (496h37m1.904188116s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:06:02.033140599 (496h37m1.904223654s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:06:02.328202557 (496h37m2.199285602s) [upload=435884]: Done uploading segments... Dec 26 08:06:02.328232714 (496h37m2.199315759s) [upload=435884]: Upload finished. err: <nil> Duration: 6.046457345s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)