Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XX.X.........X........................X............X.] duration: 5.679576863s trace: https://jaeger.dev.storj.io/trace/581400723f24346f FAIL (2025-12-23 15:48:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 7PT7YT4LUUEGU6FXRTSA554ZSBCU2GMDDVDE2JBYXCUM7QQXMWQA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: TN4U3QCWEFZ4X43CJNEYX5ZXRPB7B5OMODGBF25HG6QM2G2PES2Q) context deadline exceeded ======LOGS====== Dec 23 15:47:43.011486208 (432h18m42.882569243s) [upload=379506]: Starting upload Dec 23 15:47:43.011543938 (432h18m42.882626973s) [upload=379506]: Done starting upload Dec 23 15:47:43.011637948 (432h18m42.882720993s) [upload=379506]: Uploading segments... Dec 23 15:47:43.011748107 (432h18m42.882831152s) [upload=379506]: Got next segment. Inline: false Dec 23 15:47:43.011782613 (432h18m42.882865658s) [upload=379506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:47:43.011787232 (432h18m42.882870277s) [upload=379506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:47:43.256200465 (432h18m43.1272835s) [seg_pos={0 0}, upload=379506]: Begin upload segment... Dec 23 15:47:43.256247605 (432h18m43.12733064s) [seg_pos={0 0}, upload=379506]: Done begin upload segment. Dec 23 15:47:43.256272452 (432h18m43.127355497s) [noise=true , address=145.79.227.39:4015 , piece_id=CZFPFVOIIXMMSK7C7BY3CKXAA3FVWBLXHW4B7PHJPGIWDU4LTYAA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.256279466 (432h18m43.127362501s) [noise=true , address=145.79.227.7:4293 , piece_id=C5JFX4HGE3XLC6MNY6VLPUTZ6SZ2GTXPWQZRJHIOXTVKDUJFVHKQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.256331264 (432h18m43.127414299s) [noise=true , address=77.90.31.17:4773 , piece_id=TNTINYWYX5E6RNR7I6RPHSYXETWNT2V4OTPSHCN45PO2JRQY7VBQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.256631325 (432h18m43.12771436s) [noise=true , address=145.79.227.42:4121 , piece_id=ES3XN6MW67QDSMRENINQ7RBEMBLGG3DP2KIO4H6AG55PYGWRTLXQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.256928972 (432h18m43.128012007s) [noise=true , address=145.79.227.51:4411 , piece_id=6BQWGY6OWA3UF2QUWKZMS7VXTEPEGV5M4NOVETUVP6D4OKSPN73A, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.257202873 (432h18m43.128285918s) [noise=true , address=145.79.227.34:4907 , piece_id=SOJ2WKMXIE44W5R4LUDIPRC5AHACOBUWAMML6463WK6JDYOER65Q, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.257472046 (432h18m43.128555081s) [noise=true , address=45.91.116.14:4189 , piece_id=TN4U3QCWEFZ4X43CJNEYX5ZXRPB7B5OMODGBF25HG6QM2G2PES2Q, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.257726951 (432h18m43.128809986s) [noise=true , address=145.79.227.36:4518 , piece_id=MEMDEC7K2SXKJN7SXCPJURSVPRZIFKSLGTLCA3POSN3QXS2KGXWA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.258037072 (432h18m43.129120107s) [noise=true , address=145.79.227.44:4193 , piece_id=HC5IY3WMKDNXONBJ7NYOBME6RUNZ7YRA7QSKSM35GJJELRZN4CBQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.258331804 (432h18m43.129414839s) [noise=true , address=145.79.227.48:4302 , piece_id=5YYER23IPZJMHO2FW3EC6MEHFWZNGMRCAJXZFRDCO2PTTAHDAYXQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.258628849 (432h18m43.129711884s) [noise=true , address=145.79.227.49:4338 , piece_id=GJCHSDI657VK4ISR6SX2BXRLJRKRMAQ362EAOBKKAK5NZFHDRCLA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.259006819 (432h18m43.130089854s) [noise=true , address=145.79.227.37:4567 , piece_id=5BYMV7442EKIWAAHWJZA2EPLK5CSWUKRI4LNTAEHAWEB7OMJYA2A, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.259838813 (432h18m43.130921858s) [noise=true , address=45.91.116.7:4011 , piece_id=UIHQYATS4V6Y4EP6NKEN6OGGBC5VWBDO6LACZQ3B7O7LCREVAIFA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.260911015 (432h18m43.13199405s) [noise=true , address=45.91.116.11:4120 , piece_id=7PT7YT4LUUEGU6FXRTSA554ZSBCU2GMDDVDE2JBYXCUM7QQXMWQA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.261943862 (432h18m43.133026907s) [noise=true , address=145.79.227.52:4447 , piece_id=KTFDES4FFJXKYTC42UAMQWRIQYMHV5Q4XZN63SOPJSAGN7ZKKEYA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.263944210 (432h18m43.135027245s) [noise=true , address=145.79.227.20:4438 , piece_id=VQM3E3FSDAOIIBIMF3LMA4NQFV45HXLVFTQKD4D2V7WUNEWBODLA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:44.912380269 (432h18m44.783463314s) [upload=379506]: Next returned nil segment Dec 23 15:47:44.912395739 (432h18m44.783478774s) [upload=379506]: Waiting for error group managing segments... Dec 23 15:47:45.237752803 (432h18m45.108835848s) [noise=true , address=145.79.227.49:4338 , piece_id=GJCHSDI657VK4ISR6SX2BXRLJRKRMAQ362EAOBKKAK5NZFHDRCLA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.243294205 (432h18m45.11437724s) [noise=true , address=145.79.227.52:4447 , piece_id=KTFDES4FFJXKYTC42UAMQWRIQYMHV5Q4XZN63SOPJSAGN7ZKKEYA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.245640923 (432h18m45.116723968s) [noise=true , address=145.79.227.37:4567 , piece_id=5BYMV7442EKIWAAHWJZA2EPLK5CSWUKRI4LNTAEHAWEB7OMJYA2A, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.246865004 (432h18m45.117948039s) [noise=true , address=145.79.227.7:4293 , piece_id=C5JFX4HGE3XLC6MNY6VLPUTZ6SZ2GTXPWQZRJHIOXTVKDUJFVHKQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.257000674 (432h18m45.128083719s) [noise=true , address=145.79.227.20:4438 , piece_id=VQM3E3FSDAOIIBIMF3LMA4NQFV45HXLVFTQKD4D2V7WUNEWBODLA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.257285677 (432h18m45.128368712s) [noise=true , address=145.79.227.39:4015 , piece_id=CZFPFVOIIXMMSK7C7BY3CKXAA3FVWBLXHW4B7PHJPGIWDU4LTYAA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.260196700 (432h18m45.131279735s) [noise=true , address=145.79.227.51:4411 , piece_id=6BQWGY6OWA3UF2QUWKZMS7VXTEPEGV5M4NOVETUVP6D4OKSPN73A, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.264292880 (432h18m45.135375915s) [noise=true , address=145.79.227.48:4302 , piece_id=5YYER23IPZJMHO2FW3EC6MEHFWZNGMRCAJXZFRDCO2PTTAHDAYXQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.266872141 (432h18m45.137955186s) [noise=true , address=145.79.227.34:4907 , piece_id=SOJ2WKMXIE44W5R4LUDIPRC5AHACOBUWAMML6463WK6JDYOER65Q, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.267812602 (432h18m45.138895637s) [noise=true , address=145.79.227.36:4518 , piece_id=MEMDEC7K2SXKJN7SXCPJURSVPRZIFKSLGTLCA3POSN3QXS2KGXWA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.268395542 (432h18m45.139478577s) [noise=true , address=145.79.227.42:4121 , piece_id=ES3XN6MW67QDSMRENINQ7RBEMBLGG3DP2KIO4H6AG55PYGWRTLXQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.511242934 (432h18m45.38232597s) [noise=true , address=45.91.116.11:4120 , piece_id=7PT7YT4LUUEGU6FXRTSA554ZSBCU2GMDDVDE2JBYXCUM7QQXMWQA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.518460999 (432h18m45.389544044s) [noise=true , address=45.91.116.14:4189 , piece_id=TN4U3QCWEFZ4X43CJNEYX5ZXRPB7B5OMODGBF25HG6QM2G2PES2Q, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.518482088 (432h18m45.389565133s) [seg_pos={0 0}, upload=379506]: Segment reached optimal threshold of 13 pieces. Dec 23 15:47:45.518735061 (432h18m45.389818096s) [noise=true , address=145.79.227.44:4193 , piece_id=HC5IY3WMKDNXONBJ7NYOBME6RUNZ7YRA7QSKSM35GJJELRZN4CBQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 23 15:47:45.518741332 (432h18m45.389824367s) [noise=true , address=145.79.227.44:4193 , piece_id=HC5IY3WMKDNXONBJ7NYOBME6RUNZ7YRA7QSKSM35GJJELRZN4CBQ, seg_pos={0 0}, upload=379506]: Long tail cancellation due to optimal threshold Dec 23 15:47:45.518942435 (432h18m45.390025491s) [noise=true , address=45.91.116.7:4011 , piece_id=UIHQYATS4V6Y4EP6NKEN6OGGBC5VWBDO6LACZQ3B7O7LCREVAIFA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 23 15:47:45.518958315 (432h18m45.39004135s) [noise=true , address=45.91.116.7:4011 , piece_id=UIHQYATS4V6Y4EP6NKEN6OGGBC5VWBDO6LACZQ3B7O7LCREVAIFA, seg_pos={0 0}, upload=379506]: Long tail cancellation due to optimal threshold Dec 23 15:47:45.519028689 (432h18m45.390111724s) [noise=true , address=77.90.31.17:4773 , piece_id=TNTINYWYX5E6RNR7I6RPHSYXETWNT2V4OTPSHCN45PO2JRQY7VBQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 23 15:47:45.519035893 (432h18m45.390118928s) [noise=true , address=77.90.31.17:4773 , piece_id=TNTINYWYX5E6RNR7I6RPHSYXETWNT2V4OTPSHCN45PO2JRQY7VBQ, seg_pos={0 0}, upload=379506]: Long tail cancellation due to optimal threshold Dec 23 15:47:45.519084005 (432h18m45.39016704s) [seg_pos={0 0}, upload=379506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:47:45.519145512 (432h18m45.390228547s) [upload=379506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:47:45.519151223 (432h18m45.390234268s) [upload=379506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:47:45.806536452 (432h18m45.677619488s) [upload=379506]: Done uploading segments... Dec 23 15:47:45.806551080 (432h18m45.677634125s) [upload=379506]: Upload finished. err: <nil> Duration: 5.679609275s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.386646006s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:48:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.38671147s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.403532039s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:48:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.403555363s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XX.X.........X........................X............X] duration: 30.003381409s trace: https://jaeger.dev.storj.io/trace/581400723f24346f FAIL (2025-12-23 15:48:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 7PT7YT4LUUEGU6FXRTSA554ZSBCU2GMDDVDE2JBYXCUM7QQXMWQA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: TN4U3QCWEFZ4X43CJNEYX5ZXRPB7B5OMODGBF25HG6QM2G2PES2Q) context deadline exceeded ======LOGS====== Dec 23 15:47:43.011486208 (432h18m42.882569243s) [upload=379506]: Starting upload Dec 23 15:47:43.011543938 (432h18m42.882626973s) [upload=379506]: Done starting upload Dec 23 15:47:43.011637948 (432h18m42.882720993s) [upload=379506]: Uploading segments... Dec 23 15:47:43.011748107 (432h18m42.882831152s) [upload=379506]: Got next segment. Inline: false Dec 23 15:47:43.011782613 (432h18m42.882865658s) [upload=379506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:47:43.011787232 (432h18m42.882870277s) [upload=379506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:47:43.256200465 (432h18m43.1272835s) [seg_pos={0 0}, upload=379506]: Begin upload segment... Dec 23 15:47:43.256247605 (432h18m43.12733064s) [seg_pos={0 0}, upload=379506]: Done begin upload segment. Dec 23 15:47:43.256272452 (432h18m43.127355497s) [noise=true , address=145.79.227.39:4015 , piece_id=CZFPFVOIIXMMSK7C7BY3CKXAA3FVWBLXHW4B7PHJPGIWDU4LTYAA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.256279466 (432h18m43.127362501s) [noise=true , address=145.79.227.7:4293 , piece_id=C5JFX4HGE3XLC6MNY6VLPUTZ6SZ2GTXPWQZRJHIOXTVKDUJFVHKQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.256331264 (432h18m43.127414299s) [noise=true , address=77.90.31.17:4773 , piece_id=TNTINYWYX5E6RNR7I6RPHSYXETWNT2V4OTPSHCN45PO2JRQY7VBQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.256631325 (432h18m43.12771436s) [noise=true , address=145.79.227.42:4121 , piece_id=ES3XN6MW67QDSMRENINQ7RBEMBLGG3DP2KIO4H6AG55PYGWRTLXQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.256928972 (432h18m43.128012007s) [noise=true , address=145.79.227.51:4411 , piece_id=6BQWGY6OWA3UF2QUWKZMS7VXTEPEGV5M4NOVETUVP6D4OKSPN73A, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.257202873 (432h18m43.128285918s) [noise=true , address=145.79.227.34:4907 , piece_id=SOJ2WKMXIE44W5R4LUDIPRC5AHACOBUWAMML6463WK6JDYOER65Q, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.257472046 (432h18m43.128555081s) [noise=true , address=45.91.116.14:4189 , piece_id=TN4U3QCWEFZ4X43CJNEYX5ZXRPB7B5OMODGBF25HG6QM2G2PES2Q, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.257726951 (432h18m43.128809986s) [noise=true , address=145.79.227.36:4518 , piece_id=MEMDEC7K2SXKJN7SXCPJURSVPRZIFKSLGTLCA3POSN3QXS2KGXWA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.258037072 (432h18m43.129120107s) [noise=true , address=145.79.227.44:4193 , piece_id=HC5IY3WMKDNXONBJ7NYOBME6RUNZ7YRA7QSKSM35GJJELRZN4CBQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.258331804 (432h18m43.129414839s) [noise=true , address=145.79.227.48:4302 , piece_id=5YYER23IPZJMHO2FW3EC6MEHFWZNGMRCAJXZFRDCO2PTTAHDAYXQ, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.258628849 (432h18m43.129711884s) [noise=true , address=145.79.227.49:4338 , piece_id=GJCHSDI657VK4ISR6SX2BXRLJRKRMAQ362EAOBKKAK5NZFHDRCLA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.259006819 (432h18m43.130089854s) [noise=true , address=145.79.227.37:4567 , piece_id=5BYMV7442EKIWAAHWJZA2EPLK5CSWUKRI4LNTAEHAWEB7OMJYA2A, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.259838813 (432h18m43.130921858s) [noise=true , address=45.91.116.7:4011 , piece_id=UIHQYATS4V6Y4EP6NKEN6OGGBC5VWBDO6LACZQ3B7O7LCREVAIFA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.260911015 (432h18m43.13199405s) [noise=true , address=45.91.116.11:4120 , piece_id=7PT7YT4LUUEGU6FXRTSA554ZSBCU2GMDDVDE2JBYXCUM7QQXMWQA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.261943862 (432h18m43.133026907s) [noise=true , address=145.79.227.52:4447 , piece_id=KTFDES4FFJXKYTC42UAMQWRIQYMHV5Q4XZN63SOPJSAGN7ZKKEYA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:43.263944210 (432h18m43.135027245s) [noise=true , address=145.79.227.20:4438 , piece_id=VQM3E3FSDAOIIBIMF3LMA4NQFV45HXLVFTQKD4D2V7WUNEWBODLA, seg_pos={0 0}, upload=379506]: Uploading piece... Dec 23 15:47:44.912380269 (432h18m44.783463314s) [upload=379506]: Next returned nil segment Dec 23 15:47:44.912395739 (432h18m44.783478774s) [upload=379506]: Waiting for error group managing segments... Dec 23 15:47:45.237752803 (432h18m45.108835848s) [noise=true , address=145.79.227.49:4338 , piece_id=GJCHSDI657VK4ISR6SX2BXRLJRKRMAQ362EAOBKKAK5NZFHDRCLA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.243294205 (432h18m45.11437724s) [noise=true , address=145.79.227.52:4447 , piece_id=KTFDES4FFJXKYTC42UAMQWRIQYMHV5Q4XZN63SOPJSAGN7ZKKEYA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.245640923 (432h18m45.116723968s) [noise=true , address=145.79.227.37:4567 , piece_id=5BYMV7442EKIWAAHWJZA2EPLK5CSWUKRI4LNTAEHAWEB7OMJYA2A, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.246865004 (432h18m45.117948039s) [noise=true , address=145.79.227.7:4293 , piece_id=C5JFX4HGE3XLC6MNY6VLPUTZ6SZ2GTXPWQZRJHIOXTVKDUJFVHKQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.257000674 (432h18m45.128083719s) [noise=true , address=145.79.227.20:4438 , piece_id=VQM3E3FSDAOIIBIMF3LMA4NQFV45HXLVFTQKD4D2V7WUNEWBODLA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.257285677 (432h18m45.128368712s) [noise=true , address=145.79.227.39:4015 , piece_id=CZFPFVOIIXMMSK7C7BY3CKXAA3FVWBLXHW4B7PHJPGIWDU4LTYAA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.260196700 (432h18m45.131279735s) [noise=true , address=145.79.227.51:4411 , piece_id=6BQWGY6OWA3UF2QUWKZMS7VXTEPEGV5M4NOVETUVP6D4OKSPN73A, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.264292880 (432h18m45.135375915s) [noise=true , address=145.79.227.48:4302 , piece_id=5YYER23IPZJMHO2FW3EC6MEHFWZNGMRCAJXZFRDCO2PTTAHDAYXQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.266872141 (432h18m45.137955186s) [noise=true , address=145.79.227.34:4907 , piece_id=SOJ2WKMXIE44W5R4LUDIPRC5AHACOBUWAMML6463WK6JDYOER65Q, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.267812602 (432h18m45.138895637s) [noise=true , address=145.79.227.36:4518 , piece_id=MEMDEC7K2SXKJN7SXCPJURSVPRZIFKSLGTLCA3POSN3QXS2KGXWA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.268395542 (432h18m45.139478577s) [noise=true , address=145.79.227.42:4121 , piece_id=ES3XN6MW67QDSMRENINQ7RBEMBLGG3DP2KIO4H6AG55PYGWRTLXQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.511242934 (432h18m45.38232597s) [noise=true , address=45.91.116.11:4120 , piece_id=7PT7YT4LUUEGU6FXRTSA554ZSBCU2GMDDVDE2JBYXCUM7QQXMWQA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.518460999 (432h18m45.389544044s) [noise=true , address=45.91.116.14:4189 , piece_id=TN4U3QCWEFZ4X43CJNEYX5ZXRPB7B5OMODGBF25HG6QM2G2PES2Q, seg_pos={0 0}, upload=379506]: Done uploading piece. err: <nil> Dec 23 15:47:45.518482088 (432h18m45.389565133s) [seg_pos={0 0}, upload=379506]: Segment reached optimal threshold of 13 pieces. Dec 23 15:47:45.518735061 (432h18m45.389818096s) [noise=true , address=145.79.227.44:4193 , piece_id=HC5IY3WMKDNXONBJ7NYOBME6RUNZ7YRA7QSKSM35GJJELRZN4CBQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 23 15:47:45.518741332 (432h18m45.389824367s) [noise=true , address=145.79.227.44:4193 , piece_id=HC5IY3WMKDNXONBJ7NYOBME6RUNZ7YRA7QSKSM35GJJELRZN4CBQ, seg_pos={0 0}, upload=379506]: Long tail cancellation due to optimal threshold Dec 23 15:47:45.518942435 (432h18m45.390025491s) [noise=true , address=45.91.116.7:4011 , piece_id=UIHQYATS4V6Y4EP6NKEN6OGGBC5VWBDO6LACZQ3B7O7LCREVAIFA, seg_pos={0 0}, upload=379506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 23 15:47:45.518958315 (432h18m45.39004135s) [noise=true , address=45.91.116.7:4011 , piece_id=UIHQYATS4V6Y4EP6NKEN6OGGBC5VWBDO6LACZQ3B7O7LCREVAIFA, seg_pos={0 0}, upload=379506]: Long tail cancellation due to optimal threshold Dec 23 15:47:45.519028689 (432h18m45.390111724s) [noise=true , address=77.90.31.17:4773 , piece_id=TNTINYWYX5E6RNR7I6RPHSYXETWNT2V4OTPSHCN45PO2JRQY7VBQ, seg_pos={0 0}, upload=379506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 23 15:47:45.519035893 (432h18m45.390118928s) [noise=true , address=77.90.31.17:4773 , piece_id=TNTINYWYX5E6RNR7I6RPHSYXETWNT2V4OTPSHCN45PO2JRQY7VBQ, seg_pos={0 0}, upload=379506]: Long tail cancellation due to optimal threshold Dec 23 15:47:45.519084005 (432h18m45.39016704s) [seg_pos={0 0}, upload=379506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:47:45.519145512 (432h18m45.390228547s) [upload=379506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:47:45.519151223 (432h18m45.390234268s) [upload=379506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:47:45.806536452 (432h18m45.677619488s) [upload=379506]: Done uploading segments... Dec 23 15:47:45.806551080 (432h18m45.677634125s) [upload=379506]: Upload finished. err: <nil> Duration: 30.0034034s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X....................X....X...............X..........] duration: 5.339963365s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:48:07 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.339995826s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X....................X....X...............X.........] duration: 5.557700121s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:47:51 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.557798838s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.373029668s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:47:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.373047892s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X....................X....X...............X........] duration: 5.668828288s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:47:06 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.668856562s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XX.X.........X........................X............] duration: 5.616521939s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:46:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.616547027s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X....................X....X...............X.......] duration: 5.184657957s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:46:38 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.18468598s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.35719645s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:46:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.357231928s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.39174226s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:45:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.391772187s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X....................X....X...............X......] duration: 5.591545558s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:45:49 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.591569754s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XX.X.........X........................X...........] duration: 5.666884544s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:45:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.666920251s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.393187442s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:45:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.393277846s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XX.X.........X........................X..........] duration: 6.175924892s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:45:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 6.175959971s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X....................X....X...............X.....] duration: 5.350257775s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:44:59 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.350285818s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XX.X.........X........................X.........] duration: 5.976865371s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:44:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.976896812s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.387715231s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:44:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.387743675s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X....................X....X...............X....] duration: 5.584216555s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:43:59 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.584242334s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.40169898s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:43:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.401727583s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XX.X.........X........................X........] duration: 5.695791709s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:43:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.695816468s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X....................X....X...............X...] duration: 5.172361683s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:43:28 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.17240268s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.484292189s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:43:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.484316676s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.............XX.X.........X........................X.......] duration: 5.616656434s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:42:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.616726378s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.396675824s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:42:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.396708858s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X....................X....X...............X..] duration: 5.25432279s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:42:50 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.254350913s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X....................X....X...............X.] duration: 5.571838632s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:42:33 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.571854343s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 27.289257508s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:42:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 27.28928907s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X....................X....X...............X] duration: 30.009036955s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 15:42:27 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 30.009062714s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.............XX.X.........X........................X......] duration: 5.662912454s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:42:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.662945637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.............XX.X.........X........................X.....] duration: 5.701411542s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:42:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.701439506s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.............XX.X.........X........................X....] duration: 5.975884163s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:41:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.97591381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.............XX.X.........X........................X...] duration: 20.600188422s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:41:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 20.600217277s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X....................X....X...............] duration: 4.948630302s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:41:12 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 4.948661519s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.359816468s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:41:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.359845113s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.............XX.X.........X........................X..] duration: 6.699893166s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:40:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 6.699935708s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X....................X....X..............] duration: 5.503302288s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:40:47 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.503324932s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.378248196s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:40:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.37828247s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.............XX.X.........X........................X.] duration: 5.901807427s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:40:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 5.901854156s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.492593947s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:40:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.492620929s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.............XX.X.........X........................X] duration: 30.002736214s trace: https://jaeger.dev.storj.io/trace/7459376ebb0df750 FAIL (2025-12-23 15:40:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 15:39:43.862366650 (432h10m43.733449685s) [upload=379380]: Starting upload Dec 23 15:39:43.862427396 (432h10m43.733510441s) [upload=379380]: Done starting upload Dec 23 15:39:43.862509131 (432h10m43.733592176s) [upload=379380]: Uploading segments... Dec 23 15:39:43.862813260 (432h10m43.733896305s) [upload=379380]: Got next segment. Inline: false Dec 23 15:39:43.862874537 (432h10m43.733957572s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:39:43.862879066 (432h10m43.733962101s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:39:44.104361649 (432h10m43.975444714s) [seg_pos={0 0}, upload=379380]: Begin upload segment... Dec 23 15:39:44.104428998 (432h10m43.975512033s) [seg_pos={0 0}, upload=379380]: Done begin upload segment. Dec 23 15:39:44.104476708 (432h10m43.975559743s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.104541542 (432h10m43.975624587s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105199965 (432h10m43.976283021s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.105639321 (432h10m43.976722367s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106134785 (432h10m43.97721783s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.106616412 (432h10m43.977699457s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.107221354 (432h10m43.9783044s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.108989571 (432h10m43.980072617s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.110922872 (432h10m43.982005918s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.112851034 (432h10m43.983934079s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.114063312 (432h10m43.985146357s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115655063 (432h10m43.986738099s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.115932061 (432h10m43.987015106s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.117683696 (432h10m43.988766741s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.118822525 (432h10m43.98990557s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:44.119393462 (432h10m43.990476507s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Uploading piece... Dec 23 15:39:45.244407057 (432h10m45.115490102s) [noise=true , address=77.90.31.12:4589 , piece_id=BEQL53WGFNISE5HUFEA4AGDDLOLZJB6YOXZ4HM62HXTFGBVAMJRA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 15:39:45.723589020 (432h10m45.594672055s) [upload=379380]: Next returned nil segment Dec 23 15:39:45.723601864 (432h10m45.594684899s) [upload=379380]: Waiting for error group managing segments... Dec 23 15:39:46.104108863 (432h10m45.975191898s) [noise=true , address=145.79.227.25:4617 , piece_id=262RRZECVUA6ACLCLBE2I33TIZSVT6YWMHZFJ36QUVFHFILWYSGA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.107682338 (432h10m45.978765373s) [noise=true , address=145.79.227.41:4085 , piece_id=PHLEV2JNTN2MQYTWJ5UP536QNR7PTGUF5RBZ5GXPPJEVOHIRBT2A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.108259596 (432h10m45.979342641s) [noise=true , address=145.79.227.11:4731 , piece_id=BQZJEXLPV7562RC3J2UX5CGYZJJG5AIB2SZ3PPIITPMHF3YJFI5A, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110728147 (432h10m45.981811182s) [noise=true , address=145.79.227.2:4113 , piece_id=XSXT6P6YNYRXOA2YN3RCSPZNWUFYMWHVFD5YJWS2DMEA63YVERSQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.110838076 (432h10m45.981921111s) [noise=true , address=145.79.227.36:4517 , piece_id=KOPEUNNUQH6QLLPMX5LA2P7DUGLH3BCKQL7FT4N35HWLB7TAE2QQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.113333799 (432h10m45.984416844s) [noise=true , address=145.79.227.46:4601 , piece_id=77WQI4DFWGZCYQS3X5YKBFX2NFQG346FS7XYQIVT7XSM6VOHLEUA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.116949674 (432h10m45.988032719s) [noise=true , address=145.79.227.17:4970 , piece_id=5NUJPZFNLBJYFMXPZHBK2ML5OGVC3XDH6GZRAMMRYDKFUYNW4CIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.119743003 (432h10m45.990826038s) [noise=true , address=145.79.227.32:4834 , piece_id=TVHFSWEWV47DVXAJ6SQUHQC3D4LB4PEKU6XAB3VSX2FHR24WTGCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.122359395 (432h10m45.99344243s) [noise=true , address=145.79.227.16:4910 , piece_id=PFIV57EAV5TWIVY2WACSHR7RBYCT6TCZXHU2GECIX7Z5QMUXVTZA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.126274691 (432h10m45.997357736s) [noise=true , address=145.79.227.48:4301 , piece_id=EQSEFS4V52AOASOYERRGA6REQU4OESMTTORA6KIV7FKMHZTMJACA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.129580776 (432h10m46.000663811s) [noise=true , address=145.79.227.35:4481 , piece_id=6VXDSESLDDNHSN4O33X3RA4III7S7CZMMWCEVWRX2WV67UAGNIIA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:46.728391973 (432h10m46.599475008s) [noise=true , address=77.90.31.13:4627 , piece_id=ICQFYKW7KKXFCW63HQWNCPZFC6WLOZ7J5B3LSEJFL46VKI7SI2BA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204670978 (432h10m47.075754033s) [noise=true , address=45.91.116.7:4012 , piece_id=OBHZK5YH5XQ335ROGM56MHL4Z7WUISDBDY4MKV2J6OQ7DOWEPYPQ, seg_pos={0 0}, upload=379380]: Done uploading piece. err: <nil> Dec 23 15:39:47.204694633 (432h10m47.075777708s) [seg_pos={0 0}, upload=379380]: Segment reached optimal threshold of 13 pieces. Dec 23 15:39:47.204951782 (432h10m47.076034827s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.204961130 (432h10m47.076044175s) [noise=true , address=145.79.227.44:4194 , piece_id=EXNR2TIBAOBG3GI6YWZOHUUJF7X7KJ5FJHANT4MHZDQ6EPGY5TYA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205002640 (432h10m47.076085675s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 23 15:39:47.205012147 (432h10m47.076095193s) [noise=true , address=45.91.116.8:4046 , piece_id=OTKVBFRBGTYJ24LMVEB5FZFJ6PSS7VVFO5SL75TYWT2VDDXZLWCA, seg_pos={0 0}, upload=379380]: Long tail cancellation due to optimal threshold Dec 23 15:39:47.205036544 (432h10m47.076119579s) [seg_pos={0 0}, upload=379380]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:39:47.205116606 (432h10m47.076199641s) [upload=379380]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:39:47.205122117 (432h10m47.076205152s) [upload=379380]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:39:47.496814578 (432h10m47.367897613s) [upload=379380]: Done uploading segments... Dec 23 15:39:47.496857590 (432h10m47.367940635s) [upload=379380]: Upload finished. err: <nil> Duration: 30.002756034s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X....................X....X.............] duration: 5.870769566s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:40:01 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.870800236s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.391641758s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:39:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.391667977s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......X.............XX.X.........X........................] duration: 6.054469884s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:39:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 6.054495082s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X....................X....X............] duration: 5.363454343s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:38:53 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.363482287s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.35818923s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:38:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.358211401s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.......X.............XX.X.........X.......................] duration: 5.968541814s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:38:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.968568753s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.399469855s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:38:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.399499151s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.598297034s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:38:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.598340246s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)