Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........................X.X.X.X.XX....X.X.X.......X..XX.XX.] duration: 6.124411785s trace: https://jaeger.dev.storj.io/trace/5f324224e2d6014c FAIL (2025-12-26 23:40:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: ZNCWKORHC2HET6UEDU4BYZ5GANCGWW4ISYX73KJT4KEK5HE22A5Q) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: PNXVWKO7MIRQJSACOIKD4KYJ2YISYEOZAJCWOQNNUUNR32QLDXHQ) context deadline exceeded ======LOGS====== c 26 23:40:10.181697335 (512h11m10.05278038s) [upload=449432]: Starting upload Dec 26 23:40:10.181776015 (512h11m10.05285907s) [upload=449432]: Done starting upload Dec 26 23:40:10.182060557 (512h11m10.053143592s) [upload=449432]: Uploading segments... Dec 26 23:40:10.182168462 (512h11m10.053251507s) [upload=449432]: Got next segment. Inline: false Dec 26 23:40:10.182217585 (512h11m10.05330063s) [upload=449432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:40:10.182227203 (512h11m10.053310248s) [upload=449432]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:40:10.428448126 (512h11m10.299531161s) [seg_pos={0 0}, upload=449432]: Begin upload segment... Dec 26 23:40:10.428501307 (512h11m10.299584342s) [seg_pos={0 0}, upload=449432]: Done begin upload segment. Dec 26 23:40:10.428528328 (512h11m10.299611373s) [noise=true , address=145.79.227.15:4874 , piece_id=HPWQLXPWMO2WSH7CZJYPMKNWNVDP2LDEUKTDRILLE64PUISWXN3Q, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.429463828 (512h11m10.300546874s) [noise=true , address=145.79.227.48:4301 , piece_id=JX2CAXGSKSNHTNJLQ5NT3K6T2VGIUUD7LVQWRCQ6XDROFY4MRZPA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.430121902 (512h11m10.301204937s) [noise=true , address=145.79.227.42:4123 , piece_id=EBGMWDY4VLV3YW6LY5Q6MAFN3UGHINOQDX5T2WF2AH22OWRGE4GQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.430713708 (512h11m10.301796743s) [noise=true , address=45.91.116.8:4048 , piece_id=C76ST5AIA7L32VZPVBUFSCAEPL4JATDR2T6MJN7KSGBD2TRCPE4A, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.431164977 (512h11m10.302248022s) [noise=true , address=145.79.227.6:4259 , piece_id=N7EYHLTF4QS4WIUASDVNDAJ27DQ4FAXS7CD5Q5URCHBPLLQ6G5CQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.431559628 (512h11m10.302642673s) [noise=true , address=77.90.31.21:4880 , piece_id=XCF37ZFMXOFBTRUQCTLS6GVPTUB54XAF4NWOEL65GR45DXED5MGA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.432115485 (512h11m10.303198531s) [noise=true , address=45.91.116.14:4190 , piece_id=ZNCWKORHC2HET6UEDU4BYZ5GANCGWW4ISYX73KJT4KEK5HE22A5Q, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.432568198 (512h11m10.303651243s) [noise=true , address=145.79.227.26:4653 , piece_id=C7HW5PVDRGPVUP3J63U2W4U4D6YK5PFIE22KABDYCC5CW6B2LEAA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.433139715 (512h11m10.30422275s) [noise=true , address=77.90.31.17:4773 , piece_id=U6KJQZVM2A4BHGYNY5C5DBT46B5Z3MA5KMAR3Z6U7BJ2ZSNRMUKQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.433570415 (512h11m10.30465346s) [noise=true , address=45.91.116.13:4156 , piece_id=JQC7EXSVNBYBR6UTBK6SAICBG54W6OFOOKUVHGAOU57NMWJPBSPA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.434730022 (512h11m10.305813097s) [noise=true , address=45.91.116.5:4298 , piece_id=PNXVWKO7MIRQJSACOIKD4KYJ2YISYEOZAJCWOQNNUUNR32QLDXHQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.435362345 (512h11m10.30644538s) [noise=true , address=145.79.227.13:4802 , piece_id=2Q74ZAYORPVLUQJWNXUIW566VTYTM47J63VUUQLD2CVGO35T3DFQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.435884529 (512h11m10.306967574s) [noise=true , address=145.79.227.11:4732 , piece_id=2MBP7FMZV73JA2VLSL6BIWVDQSZRNUN6KU2VHVPPKNUA7ANW42OQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.436499850 (512h11m10.307582885s) [noise=true , address=145.79.227.36:4517 , piece_id=PYQOQEOC6NRXGV6EUYR2HCGHCJWX2SCIQYMEZ3WCY2F3FH5FQWKQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.436962411 (512h11m10.308045456s) [noise=true , address=145.79.227.23:4545 , piece_id=5SBVBGUWZDCYO65HNYZT75SZRQCLGBRXNYTCIAITBTQQ6H3VCTVA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.437344988 (512h11m10.308428033s) [noise=true , address=145.79.227.47:4265 , piece_id=AJX5RPQGPJP7NSI26HKC7PWXNNLU5TTCTMV3UH563HLLNPII6VMQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:12.047140092 (512h11m11.918223147s) [upload=449432]: Next returned nil segment Dec 26 23:40:12.047155672 (512h11m11.918238707s) [upload=449432]: Waiting for error group managing segments... Dec 26 23:40:12.428247533 (512h11m12.299330568s) [noise=true , address=145.79.227.11:4732 , piece_id=2MBP7FMZV73JA2VLSL6BIWVDQSZRNUN6KU2VHVPPKNUA7ANW42OQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.434863955 (512h11m12.30594699s) [noise=true , address=145.79.227.42:4123 , piece_id=EBGMWDY4VLV3YW6LY5Q6MAFN3UGHINOQDX5T2WF2AH22OWRGE4GQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.436102382 (512h11m12.307185427s) [noise=true , address=145.79.227.13:4802 , piece_id=2Q74ZAYORPVLUQJWNXUIW566VTYTM47J63VUUQLD2CVGO35T3DFQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.446441029 (512h11m12.317524064s) [noise=true , address=145.79.227.26:4653 , piece_id=C7HW5PVDRGPVUP3J63U2W4U4D6YK5PFIE22KABDYCC5CW6B2LEAA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.457596440 (512h11m12.328679485s) [noise=true , address=145.79.227.6:4259 , piece_id=N7EYHLTF4QS4WIUASDVNDAJ27DQ4FAXS7CD5Q5URCHBPLLQ6G5CQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.458538112 (512h11m12.329621157s) [noise=true , address=145.79.227.15:4874 , piece_id=HPWQLXPWMO2WSH7CZJYPMKNWNVDP2LDEUKTDRILLE64PUISWXN3Q, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.459937856 (512h11m12.331020891s) [noise=true , address=145.79.227.47:4265 , piece_id=AJX5RPQGPJP7NSI26HKC7PWXNNLU5TTCTMV3UH563HLLNPII6VMQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.460266141 (512h11m12.331349186s) [noise=true , address=145.79.227.48:4301 , piece_id=JX2CAXGSKSNHTNJLQ5NT3K6T2VGIUUD7LVQWRCQ6XDROFY4MRZPA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.464011349 (512h11m12.335094394s) [noise=true , address=145.79.227.36:4517 , piece_id=PYQOQEOC6NRXGV6EUYR2HCGHCJWX2SCIQYMEZ3WCY2F3FH5FQWKQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.681530916 (512h11m12.552613951s) [noise=true , address=45.91.116.14:4190 , piece_id=ZNCWKORHC2HET6UEDU4BYZ5GANCGWW4ISYX73KJT4KEK5HE22A5Q, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.700113784 (512h11m12.571196829s) [noise=true , address=45.91.116.5:4298 , piece_id=PNXVWKO7MIRQJSACOIKD4KYJ2YISYEOZAJCWOQNNUUNR32QLDXHQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.700613886 (512h11m12.571696931s) [noise=true , address=45.91.116.8:4048 , piece_id=C76ST5AIA7L32VZPVBUFSCAEPL4JATDR2T6MJN7KSGBD2TRCPE4A, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.771668328 (512h11m12.642751363s) [noise=true , address=45.91.116.13:4156 , piece_id=JQC7EXSVNBYBR6UTBK6SAICBG54W6OFOOKUVHGAOU57NMWJPBSPA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.771692223 (512h11m12.642775258s) [seg_pos={0 0}, upload=449432]: Segment reached optimal threshold of 13 pieces. Dec 26 23:40:12.771891273 (512h11m12.642974308s) [noise=true , address=77.90.31.17:4773 , piece_id=U6KJQZVM2A4BHGYNY5C5DBT46B5Z3MA5KMAR3Z6U7BJ2ZSNRMUKQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 23:40:12.771898015 (512h11m12.64298105s) [noise=true , address=77.90.31.17:4773 , piece_id=U6KJQZVM2A4BHGYNY5C5DBT46B5Z3MA5KMAR3Z6U7BJ2ZSNRMUKQ, seg_pos={0 0}, upload=449432]: Long tail cancellation due to optimal threshold Dec 26 23:40:12.772002725 (512h11m12.64308577s) [noise=true , address=77.90.31.21:4880 , piece_id=XCF37ZFMXOFBTRUQCTLS6GVPTUB54XAF4NWOEL65GR45DXED5MGA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 26 23:40:12.772014066 (512h11m12.643097111s) [noise=true , address=77.90.31.21:4880 , piece_id=XCF37ZFMXOFBTRUQCTLS6GVPTUB54XAF4NWOEL65GR45DXED5MGA, seg_pos={0 0}, upload=449432]: Long tail cancellation due to optimal threshold Dec 26 23:40:12.772122302 (512h11m12.643205347s) [noise=true , address=145.79.227.23:4545 , piece_id=5SBVBGUWZDCYO65HNYZT75SZRQCLGBRXNYTCIAITBTQQ6H3VCTVA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:55482->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:55482->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55482->145.79.227.23:4545: use of closed network connection Dec 26 23:40:12.772132271 (512h11m12.643215317s) [noise=true , address=145.79.227.23:4545 , piece_id=5SBVBGUWZDCYO65HNYZT75SZRQCLGBRXNYTCIAITBTQQ6H3VCTVA, seg_pos={0 0}, upload=449432]: Long tail cancellation due to optimal threshold Dec 26 23:40:12.772155455 (512h11m12.64323849s) [seg_pos={0 0}, upload=449432]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:40:12.772238944 (512h11m12.643321989s) [upload=449432]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:40:12.772245327 (512h11m12.643328372s) [upload=449432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:40:13.070473189 (512h11m12.941556224s) [upload=449432]: Done uploading segments... Dec 26 23:40:13.070496414 (512h11m12.941579459s) [upload=449432]: Upload finished. err: <nil> Duration: 6.124427575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........................X.X.X.X.XX....X.X.X.......X..XX.XX] duration: 30.002777699s trace: https://jaeger.dev.storj.io/trace/5f324224e2d6014c FAIL (2025-12-26 23:40:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: ZNCWKORHC2HET6UEDU4BYZ5GANCGWW4ISYX73KJT4KEK5HE22A5Q) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: PNXVWKO7MIRQJSACOIKD4KYJ2YISYEOZAJCWOQNNUUNR32QLDXHQ) context deadline exceeded ======LOGS====== c 26 23:40:10.181697335 (512h11m10.05278038s) [upload=449432]: Starting upload Dec 26 23:40:10.181776015 (512h11m10.05285907s) [upload=449432]: Done starting upload Dec 26 23:40:10.182060557 (512h11m10.053143592s) [upload=449432]: Uploading segments... Dec 26 23:40:10.182168462 (512h11m10.053251507s) [upload=449432]: Got next segment. Inline: false Dec 26 23:40:10.182217585 (512h11m10.05330063s) [upload=449432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:40:10.182227203 (512h11m10.053310248s) [upload=449432]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:40:10.428448126 (512h11m10.299531161s) [seg_pos={0 0}, upload=449432]: Begin upload segment... Dec 26 23:40:10.428501307 (512h11m10.299584342s) [seg_pos={0 0}, upload=449432]: Done begin upload segment. Dec 26 23:40:10.428528328 (512h11m10.299611373s) [noise=true , address=145.79.227.15:4874 , piece_id=HPWQLXPWMO2WSH7CZJYPMKNWNVDP2LDEUKTDRILLE64PUISWXN3Q, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.429463828 (512h11m10.300546874s) [noise=true , address=145.79.227.48:4301 , piece_id=JX2CAXGSKSNHTNJLQ5NT3K6T2VGIUUD7LVQWRCQ6XDROFY4MRZPA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.430121902 (512h11m10.301204937s) [noise=true , address=145.79.227.42:4123 , piece_id=EBGMWDY4VLV3YW6LY5Q6MAFN3UGHINOQDX5T2WF2AH22OWRGE4GQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.430713708 (512h11m10.301796743s) [noise=true , address=45.91.116.8:4048 , piece_id=C76ST5AIA7L32VZPVBUFSCAEPL4JATDR2T6MJN7KSGBD2TRCPE4A, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.431164977 (512h11m10.302248022s) [noise=true , address=145.79.227.6:4259 , piece_id=N7EYHLTF4QS4WIUASDVNDAJ27DQ4FAXS7CD5Q5URCHBPLLQ6G5CQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.431559628 (512h11m10.302642673s) [noise=true , address=77.90.31.21:4880 , piece_id=XCF37ZFMXOFBTRUQCTLS6GVPTUB54XAF4NWOEL65GR45DXED5MGA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.432115485 (512h11m10.303198531s) [noise=true , address=45.91.116.14:4190 , piece_id=ZNCWKORHC2HET6UEDU4BYZ5GANCGWW4ISYX73KJT4KEK5HE22A5Q, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.432568198 (512h11m10.303651243s) [noise=true , address=145.79.227.26:4653 , piece_id=C7HW5PVDRGPVUP3J63U2W4U4D6YK5PFIE22KABDYCC5CW6B2LEAA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.433139715 (512h11m10.30422275s) [noise=true , address=77.90.31.17:4773 , piece_id=U6KJQZVM2A4BHGYNY5C5DBT46B5Z3MA5KMAR3Z6U7BJ2ZSNRMUKQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.433570415 (512h11m10.30465346s) [noise=true , address=45.91.116.13:4156 , piece_id=JQC7EXSVNBYBR6UTBK6SAICBG54W6OFOOKUVHGAOU57NMWJPBSPA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.434730022 (512h11m10.305813097s) [noise=true , address=45.91.116.5:4298 , piece_id=PNXVWKO7MIRQJSACOIKD4KYJ2YISYEOZAJCWOQNNUUNR32QLDXHQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.435362345 (512h11m10.30644538s) [noise=true , address=145.79.227.13:4802 , piece_id=2Q74ZAYORPVLUQJWNXUIW566VTYTM47J63VUUQLD2CVGO35T3DFQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.435884529 (512h11m10.306967574s) [noise=true , address=145.79.227.11:4732 , piece_id=2MBP7FMZV73JA2VLSL6BIWVDQSZRNUN6KU2VHVPPKNUA7ANW42OQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.436499850 (512h11m10.307582885s) [noise=true , address=145.79.227.36:4517 , piece_id=PYQOQEOC6NRXGV6EUYR2HCGHCJWX2SCIQYMEZ3WCY2F3FH5FQWKQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.436962411 (512h11m10.308045456s) [noise=true , address=145.79.227.23:4545 , piece_id=5SBVBGUWZDCYO65HNYZT75SZRQCLGBRXNYTCIAITBTQQ6H3VCTVA, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:10.437344988 (512h11m10.308428033s) [noise=true , address=145.79.227.47:4265 , piece_id=AJX5RPQGPJP7NSI26HKC7PWXNNLU5TTCTMV3UH563HLLNPII6VMQ, seg_pos={0 0}, upload=449432]: Uploading piece... Dec 26 23:40:12.047140092 (512h11m11.918223147s) [upload=449432]: Next returned nil segment Dec 26 23:40:12.047155672 (512h11m11.918238707s) [upload=449432]: Waiting for error group managing segments... Dec 26 23:40:12.428247533 (512h11m12.299330568s) [noise=true , address=145.79.227.11:4732 , piece_id=2MBP7FMZV73JA2VLSL6BIWVDQSZRNUN6KU2VHVPPKNUA7ANW42OQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.434863955 (512h11m12.30594699s) [noise=true , address=145.79.227.42:4123 , piece_id=EBGMWDY4VLV3YW6LY5Q6MAFN3UGHINOQDX5T2WF2AH22OWRGE4GQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.436102382 (512h11m12.307185427s) [noise=true , address=145.79.227.13:4802 , piece_id=2Q74ZAYORPVLUQJWNXUIW566VTYTM47J63VUUQLD2CVGO35T3DFQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.446441029 (512h11m12.317524064s) [noise=true , address=145.79.227.26:4653 , piece_id=C7HW5PVDRGPVUP3J63U2W4U4D6YK5PFIE22KABDYCC5CW6B2LEAA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.457596440 (512h11m12.328679485s) [noise=true , address=145.79.227.6:4259 , piece_id=N7EYHLTF4QS4WIUASDVNDAJ27DQ4FAXS7CD5Q5URCHBPLLQ6G5CQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.458538112 (512h11m12.329621157s) [noise=true , address=145.79.227.15:4874 , piece_id=HPWQLXPWMO2WSH7CZJYPMKNWNVDP2LDEUKTDRILLE64PUISWXN3Q, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.459937856 (512h11m12.331020891s) [noise=true , address=145.79.227.47:4265 , piece_id=AJX5RPQGPJP7NSI26HKC7PWXNNLU5TTCTMV3UH563HLLNPII6VMQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.460266141 (512h11m12.331349186s) [noise=true , address=145.79.227.48:4301 , piece_id=JX2CAXGSKSNHTNJLQ5NT3K6T2VGIUUD7LVQWRCQ6XDROFY4MRZPA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.464011349 (512h11m12.335094394s) [noise=true , address=145.79.227.36:4517 , piece_id=PYQOQEOC6NRXGV6EUYR2HCGHCJWX2SCIQYMEZ3WCY2F3FH5FQWKQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.681530916 (512h11m12.552613951s) [noise=true , address=45.91.116.14:4190 , piece_id=ZNCWKORHC2HET6UEDU4BYZ5GANCGWW4ISYX73KJT4KEK5HE22A5Q, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.700113784 (512h11m12.571196829s) [noise=true , address=45.91.116.5:4298 , piece_id=PNXVWKO7MIRQJSACOIKD4KYJ2YISYEOZAJCWOQNNUUNR32QLDXHQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.700613886 (512h11m12.571696931s) [noise=true , address=45.91.116.8:4048 , piece_id=C76ST5AIA7L32VZPVBUFSCAEPL4JATDR2T6MJN7KSGBD2TRCPE4A, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.771668328 (512h11m12.642751363s) [noise=true , address=45.91.116.13:4156 , piece_id=JQC7EXSVNBYBR6UTBK6SAICBG54W6OFOOKUVHGAOU57NMWJPBSPA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: <nil> Dec 26 23:40:12.771692223 (512h11m12.642775258s) [seg_pos={0 0}, upload=449432]: Segment reached optimal threshold of 13 pieces. Dec 26 23:40:12.771891273 (512h11m12.642974308s) [noise=true , address=77.90.31.17:4773 , piece_id=U6KJQZVM2A4BHGYNY5C5DBT46B5Z3MA5KMAR3Z6U7BJ2ZSNRMUKQ, seg_pos={0 0}, upload=449432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 23:40:12.771898015 (512h11m12.64298105s) [noise=true , address=77.90.31.17:4773 , piece_id=U6KJQZVM2A4BHGYNY5C5DBT46B5Z3MA5KMAR3Z6U7BJ2ZSNRMUKQ, seg_pos={0 0}, upload=449432]: Long tail cancellation due to optimal threshold Dec 26 23:40:12.772002725 (512h11m12.64308577s) [noise=true , address=77.90.31.21:4880 , piece_id=XCF37ZFMXOFBTRUQCTLS6GVPTUB54XAF4NWOEL65GR45DXED5MGA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 26 23:40:12.772014066 (512h11m12.643097111s) [noise=true , address=77.90.31.21:4880 , piece_id=XCF37ZFMXOFBTRUQCTLS6GVPTUB54XAF4NWOEL65GR45DXED5MGA, seg_pos={0 0}, upload=449432]: Long tail cancellation due to optimal threshold Dec 26 23:40:12.772122302 (512h11m12.643205347s) [noise=true , address=145.79.227.23:4545 , piece_id=5SBVBGUWZDCYO65HNYZT75SZRQCLGBRXNYTCIAITBTQQ6H3VCTVA, seg_pos={0 0}, upload=449432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:55482->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:55482->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55482->145.79.227.23:4545: use of closed network connection Dec 26 23:40:12.772132271 (512h11m12.643215317s) [noise=true , address=145.79.227.23:4545 , piece_id=5SBVBGUWZDCYO65HNYZT75SZRQCLGBRXNYTCIAITBTQQ6H3VCTVA, seg_pos={0 0}, upload=449432]: Long tail cancellation due to optimal threshold Dec 26 23:40:12.772155455 (512h11m12.64323849s) [seg_pos={0 0}, upload=449432]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:40:12.772238944 (512h11m12.643321989s) [upload=449432]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:40:12.772245327 (512h11m12.643328372s) [upload=449432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:40:13.070473189 (512h11m12.941556224s) [upload=449432]: Done uploading segments... Dec 26 23:40:13.070496414 (512h11m12.941579459s) [upload=449432]: Upload finished. err: <nil> Duration: 30.002806304s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.918218259s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:40:32 UTC) orbiter warning: context deadline exceeded Duration: 5.918243336s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................X.X.X.X.XX....X.X.X.......X..XX.X] duration: 30.001881076s trace: https://jaeger.dev.storj.io/trace/31b07965e4f43e63 FAIL (2025-12-26 23:39:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FJNHJS5IXJ2ICTVAUUFW6KNXNNBTRWLM7MJXQQCXGN6X7PB75XKQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: C3RE7CQTCFOER5TFR3HXTWVOO337NKWONVONMJFNWNOAW227FBMQ) context deadline exceeded ======LOGS====== egments... Dec 26 23:39:22.910363887 (512h10m22.781446922s) [upload=449419]: Got next segment. Inline: false Dec 26 23:39:22.910404314 (512h10m22.781487349s) [upload=449419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:39:22.910410205 (512h10m22.78149324s) [upload=449419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:39:23.158459898 (512h10m23.029542943s) [seg_pos={0 0}, upload=449419]: Begin upload segment... Dec 26 23:39:23.158518729 (512h10m23.029601774s) [seg_pos={0 0}, upload=449419]: Done begin upload segment. Dec 26 23:39:23.158554277 (512h10m23.029637322s) [noise=true , address=77.90.31.15:4700 , piece_id=C3RE7CQTCFOER5TFR3HXTWVOO337NKWONVONMJFNWNOAW227FBMQ, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.158661321 (512h10m23.029744356s) [noise=true , address=45.91.116.7:4012 , piece_id=FJNHJS5IXJ2ICTVAUUFW6KNXNNBTRWLM7MJXQQCXGN6X7PB75XKQ, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.159226306 (512h10m23.030309351s) [noise=true , address=45.91.116.11:4117 , piece_id=IO36MEXMBWWRHZSQBP74W6WNIOJNUZ2WULYBR6YYQKRDDSDW76SA, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.159682996 (512h10m23.030766041s) [noise=true , address=77.90.31.10:4518 , piece_id=ZKKOBSJJCPCYYGAOHVLPICQ4W6Q72RYAEOBVCXD4KTNRGNB2JP6Q, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.160166556 (512h10m23.031249591s) [noise=true , address=145.79.227.6:4258 , piece_id=TORQXBPKDWYNKBLJHYGWL4WOZ3KZB5DJMOUE6ACV7U5YEIEUSPCQ, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.160616531 (512h10m23.031699566s) [noise=true , address=77.90.31.14:4662 , piece_id=P6245CUIZGKE576PUNKW2OBUYCBEESYJKGVFBDSOV222GVNRST4Q, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.160981807 (512h10m23.032064843s) [noise=true , address=145.79.227.5:4223 , piece_id=5OFDXUJV664HUXVGMNRHROEEGFTMQAKZJWFITLISX4YMNZHU4S5Q, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.161026422 (512h10m23.032109467s) [noise=true , address=145.79.227.13:4804 , piece_id=SCSUNLPLA3LFIN4E4EVLCITJOODCWMOCDRVGD5ETKU6IOBIPWOIA, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.161433206 (512h10m23.032516301s) [noise=true , address=145.79.227.42:4122 , piece_id=X55J24SWPKD32FDQUCBTN67STTSKOZZ466DVYDFA7RLP2VRAQU6A, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.161770357 (512h10m23.032853403s) [noise=true , address=145.79.227.22:4510 , piece_id=I74HJNCVL5TQALQF7PKOBKKTNRSLZXTWY5OZAA7MUC5GR3B4F2NA, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.162145993 (512h10m23.033229038s) [noise=true , address=145.79.227.48:4303 , piece_id=PDOZQTIJ6UFMS2V47CB2LN4VOFWS7FXKIOZJDC2MKFU3QNS2VREA, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.162819144 (512h10m23.033902189s) [noise=true , address=145.79.227.12:4767 , piece_id=PROQZVXRNTKEWG6KIHFJPUE6CTSKDSJYE7KWUCAH56AGV52T5QWA, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.163493417 (512h10m23.034576452s) [noise=true , address=145.79.227.15:4873 , piece_id=PVTJTUCWYS5PTOJCDIDHLDH7NROBQLXWHDKKOT6KXPKB7T7OTIJQ, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.163983029 (512h10m23.035066074s) [noise=true , address=45.91.116.6:4974 , piece_id=VFPZKXK6B3RTRG734OFJBHZAAGJPSKPFYVU6S52FA4LJQ4RNHH2Q, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.164674966 (512h10m23.035758001s) [noise=true , address=77.90.31.12:4589 , piece_id=53IZQHIEK5XDMQXHA27OXQYBDNR4DP77DV7N4IMET4XJ53IIDLQA, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:23.165166201 (512h10m23.036249236s) [noise=true , address=145.79.227.19:4077 , piece_id=NG6MRR3TJFRSKK27I6ZVAOVC32NV7PZLKB6HS4O6Z7GNRBIO6S7A, seg_pos={0 0}, upload=449419]: Uploading piece... Dec 26 23:39:24.875796595 (512h10m24.74687963s) [upload=449419]: Next returned nil segment Dec 26 23:39:24.875808348 (512h10m24.746891383s) [upload=449419]: Waiting for error group managing segments... Dec 26 23:39:25.160284127 (512h10m25.031367162s) [noise=true , address=145.79.227.22:4510 , piece_id=I74HJNCVL5TQALQF7PKOBKKTNRSLZXTWY5OZAA7MUC5GR3B4F2NA, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.163881133 (512h10m25.034964168s) [noise=true , address=145.79.227.42:4122 , piece_id=X55J24SWPKD32FDQUCBTN67STTSKOZZ466DVYDFA7RLP2VRAQU6A, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.168679455 (512h10m25.03976249s) [noise=true , address=145.79.227.6:4258 , piece_id=TORQXBPKDWYNKBLJHYGWL4WOZ3KZB5DJMOUE6ACV7U5YEIEUSPCQ, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.171367742 (512h10m25.042450777s) [noise=true , address=145.79.227.13:4804 , piece_id=SCSUNLPLA3LFIN4E4EVLCITJOODCWMOCDRVGD5ETKU6IOBIPWOIA, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.178726377 (512h10m25.049809412s) [noise=true , address=145.79.227.19:4077 , piece_id=NG6MRR3TJFRSKK27I6ZVAOVC32NV7PZLKB6HS4O6Z7GNRBIO6S7A, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.180504962 (512h10m25.051587998s) [noise=true , address=145.79.227.15:4873 , piece_id=PVTJTUCWYS5PTOJCDIDHLDH7NROBQLXWHDKKOT6KXPKB7T7OTIJQ, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.183986788 (512h10m25.055069823s) [noise=true , address=145.79.227.48:4303 , piece_id=PDOZQTIJ6UFMS2V47CB2LN4VOFWS7FXKIOZJDC2MKFU3QNS2VREA, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.205584003 (512h10m25.076667048s) [noise=true , address=145.79.227.12:4767 , piece_id=PROQZVXRNTKEWG6KIHFJPUE6CTSKDSJYE7KWUCAH56AGV52T5QWA, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.326744588 (512h10m25.197827623s) [noise=true , address=77.90.31.15:4700 , piece_id=C3RE7CQTCFOER5TFR3HXTWVOO337NKWONVONMJFNWNOAW227FBMQ, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.417342165 (512h10m25.2884252s) [noise=true , address=77.90.31.10:4518 , piece_id=ZKKOBSJJCPCYYGAOHVLPICQ4W6Q72RYAEOBVCXD4KTNRGNB2JP6Q, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.436861294 (512h10m25.307944329s) [noise=true , address=77.90.31.14:4662 , piece_id=P6245CUIZGKE576PUNKW2OBUYCBEESYJKGVFBDSOV222GVNRST4Q, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.510765651 (512h10m25.381848686s) [noise=true , address=45.91.116.11:4117 , piece_id=IO36MEXMBWWRHZSQBP74W6WNIOJNUZ2WULYBR6YYQKRDDSDW76SA, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.710298876 (512h10m25.581381921s) [noise=true , address=45.91.116.7:4012 , piece_id=FJNHJS5IXJ2ICTVAUUFW6KNXNNBTRWLM7MJXQQCXGN6X7PB75XKQ, seg_pos={0 0}, upload=449419]: Done uploading piece. err: <nil> Dec 26 23:39:25.710336949 (512h10m25.581419984s) [seg_pos={0 0}, upload=449419]: Segment reached optimal threshold of 13 pieces. Dec 26 23:39:25.710580893 (512h10m25.581663938s) [noise=true , address=45.91.116.6:4974 , piece_id=VFPZKXK6B3RTRG734OFJBHZAAGJPSKPFYVU6S52FA4LJQ4RNHH2Q, seg_pos={0 0}, upload=449419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:47238->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:47238->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47238->45.91.116.6:4974: use of closed network connection Dec 26 23:39:25.710597053 (512h10m25.581680088s) [noise=true , address=45.91.116.6:4974 , piece_id=VFPZKXK6B3RTRG734OFJBHZAAGJPSKPFYVU6S52FA4LJQ4RNHH2Q, seg_pos={0 0}, upload=449419]: Long tail cancellation due to optimal threshold Dec 26 23:39:25.710614177 (512h10m25.581697232s) [noise=true , address=145.79.227.5:4223 , piece_id=5OFDXUJV664HUXVGMNRHROEEGFTMQAKZJWFITLISX4YMNZHU4S5Q, seg_pos={0 0}, upload=449419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:54714->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:54714->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54714->145.79.227.5:4223: use of closed network connection Dec 26 23:39:25.710626149 (512h10m25.581709194s) [noise=true , address=145.79.227.5:4223 , piece_id=5OFDXUJV664HUXVGMNRHROEEGFTMQAKZJWFITLISX4YMNZHU4S5Q, seg_pos={0 0}, upload=449419]: Long tail cancellation due to optimal threshold Dec 26 23:39:25.710774952 (512h10m25.581858007s) [noise=true , address=77.90.31.12:4589 , piece_id=53IZQHIEK5XDMQXHA27OXQYBDNR4DP77DV7N4IMET4XJ53IIDLQA, seg_pos={0 0}, upload=449419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 26 23:39:25.710783458 (512h10m25.581866503s) [noise=true , address=77.90.31.12:4589 , piece_id=53IZQHIEK5XDMQXHA27OXQYBDNR4DP77DV7N4IMET4XJ53IIDLQA, seg_pos={0 0}, upload=449419]: Long tail cancellation due to optimal threshold Dec 26 23:39:25.710803857 (512h10m25.581886902s) [seg_pos={0 0}, upload=449419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:39:25.710887486 (512h10m25.581970521s) [upload=449419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:39:25.710895802 (512h10m25.581978847s) [upload=449419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:39:26.014512731 (512h10m25.885595776s) [upload=449419]: Done uploading segments... Dec 26 23:39:26.014538731 (512h10m25.885621776s) [upload=449419]: Upload finished. err: <nil> Duration: 30.001901595s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.291340368s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:39:32 UTC) orbiter warning: context deadline exceeded Duration: 6.291366517s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............................X.X.X.X.XX....X.X.X.......X..XX.] duration: 29.67339017s trace: https://jaeger.dev.storj.io/trace/5f8525aea4699e15 FAIL (2025-12-26 23:38:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: IY4SR4GQG5S3FZWGLFTYDLTJNJNSSAVTXUONL32DVYQD47D4NKHQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 5BKJYORH3KO7NZ6WZFKI5XYZS2N2HOQTZAWAIMXSWWPON7Z3AA3A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: BS5HWYDC5ULS7AZZKLV2XCWEDBQJU7V5OLDYYRLAHF7XEFO4Y7OA) context deadline exceeded ======LOGS====== Dec 26 23:37:48.433532566 (512h8m48.304615601s) [upload=449394]: Starting upload Dec 26 23:37:48.433597038 (512h8m48.304680084s) [upload=449394]: Done starting upload Dec 26 23:37:48.433763325 (512h8m48.30484636s) [upload=449394]: Uploading segments... Dec 26 23:37:48.433941715 (512h8m48.30502476s) [upload=449394]: Got next segment. Inline: false Dec 26 23:37:48.434008202 (512h8m48.305091247s) [upload=449394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:37:48.434013642 (512h8m48.305096687s) [upload=449394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:37:48.678938819 (512h8m48.550021864s) [seg_pos={0 0}, upload=449394]: Begin upload segment... Dec 26 23:37:48.678993332 (512h8m48.550076367s) [seg_pos={0 0}, upload=449394]: Done begin upload segment. Dec 26 23:37:48.679022328 (512h8m48.550105363s) [noise=true , address=77.90.31.17:4773 , piece_id=BS5HWYDC5ULS7AZZKLV2XCWEDBQJU7V5OLDYYRLAHF7XEFO4Y7OA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.679084987 (512h8m48.550168022s) [noise=true , address=77.90.31.10:4519 , piece_id=IY4SR4GQG5S3FZWGLFTYDLTJNJNSSAVTXUONL32DVYQD47D4NKHQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.679602171 (512h8m48.550685216s) [noise=true , address=145.79.227.50:4375 , piece_id=YA4O4ZBZZ7ISPINUCKF7QWPREXJGFULRGYOAJCWUHPLXZDZJ3XLQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.680053741 (512h8m48.551136786s) [noise=true , address=145.79.227.46:4603 , piece_id=NKTJ4MWPFFNAJV3PBMBVRNKEOO6PPLNEOLMG2VR2STBFTIH4BTLQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.680478409 (512h8m48.551561444s) [noise=true , address=145.79.227.45:4231 , piece_id=BITLXZF7OZOOEQQS5OD33H537XTXI4IPZO46QBREFVTC3RX4QG7A, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.680742893 (512h8m48.551825928s) [noise=true , address=77.90.31.18:4805 , piece_id=5BKJYORH3KO7NZ6WZFKI5XYZS2N2HOQTZAWAIMXSWWPON7Z3AA3A, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.681052100 (512h8m48.552135136s) [noise=true , address=45.91.116.12:4227 , piece_id=X6OFRSYOGWWHAL43G5QGHQQ6LOHO5CHJ5XUL7TRA7QNRJJQZ5CXA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.681348726 (512h8m48.552431761s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKSQ5XJI5K36RS3NPCBP4E7QP4EB7MZFUJL3TA5XW5NLA3CMPXQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.681645170 (512h8m48.552728205s) [noise=true , address=145.79.227.34:4906 , piece_id=4PUAEFIGMBKL32PDPNFNU4Y3QMQQOWJPIMDHVXBWAEKYGVNXUFYA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.683243141 (512h8m48.554326176s) [noise=true , address=145.79.227.41:4087 , piece_id=E7S2RZMLYZ3PRPT33TGFN2DCQUZ6LKS2VKZDH6RHFAJKE4OEWBTQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.684298108 (512h8m48.555381143s) [noise=true , address=145.79.227.19:4079 , piece_id=BT4BG5TZ5N5UYNXRX6L2UPXDYKX5LXRHQBYRLV7VRUR2FHBYVCNA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.685503052 (512h8m48.556586087s) [noise=true , address=145.79.227.13:4802 , piece_id=SH2YPBQOA6G6ZOQB25ZYNTNLV3TY5SFZQRUK42CJVIAPV2JMFYKA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.686990402 (512h8m48.558073438s) [noise=true , address=145.79.227.26:4653 , piece_id=R3USFFWWIEKPAIPVRGGFKUBCJVNAXHXEUTRLNO7KGQUXHKRGIVEQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.688191329 (512h8m48.559274374s) [noise=true , address=45.91.116.4:4940 , piece_id=VJJRUQMKNIUQ5U4TUSVISRUZFZYGBGMIE5BJG3KN5ILK6K3JCDOA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.691727599 (512h8m48.562810644s) [noise=true , address=145.79.227.48:4303 , piece_id=XIHPVIT3EYTA4SVYZ2STPWDSUK7NCR6MHWPVHPLOT2FMRUR5LUKA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.693260165 (512h8m48.5643432s) [noise=true , address=145.79.227.11:4732 , piece_id=K34RSS27X4CX5QMGUM4KFXO4KAD5LIXGIT3L6NPSKS5LBTJQLW2A, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:50.382876507 (512h8m50.253959552s) [upload=449394]: Next returned nil segment Dec 26 23:37:50.382894812 (512h8m50.253977857s) [upload=449394]: Waiting for error group managing segments... Dec 26 23:37:50.669447996 (512h8m50.540531031s) [noise=true , address=145.79.227.46:4603 , piece_id=NKTJ4MWPFFNAJV3PBMBVRNKEOO6PPLNEOLMG2VR2STBFTIH4BTLQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.671907436 (512h8m50.542990472s) [noise=true , address=145.79.227.45:4231 , piece_id=BITLXZF7OZOOEQQS5OD33H537XTXI4IPZO46QBREFVTC3RX4QG7A, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.678604112 (512h8m50.549687157s) [noise=true , address=145.79.227.50:4375 , piece_id=YA4O4ZBZZ7ISPINUCKF7QWPREXJGFULRGYOAJCWUHPLXZDZJ3XLQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.701564421 (512h8m50.572647456s) [noise=true , address=145.79.227.13:4802 , piece_id=SH2YPBQOA6G6ZOQB25ZYNTNLV3TY5SFZQRUK42CJVIAPV2JMFYKA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.703755230 (512h8m50.574838265s) [noise=true , address=145.79.227.48:4303 , piece_id=XIHPVIT3EYTA4SVYZ2STPWDSUK7NCR6MHWPVHPLOT2FMRUR5LUKA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.720142769 (512h8m50.591225804s) [noise=true , address=145.79.227.19:4079 , piece_id=BT4BG5TZ5N5UYNXRX6L2UPXDYKX5LXRHQBYRLV7VRUR2FHBYVCNA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.724516142 (512h8m50.595599177s) [noise=true , address=145.79.227.26:4653 , piece_id=R3USFFWWIEKPAIPVRGGFKUBCJVNAXHXEUTRLNO7KGQUXHKRGIVEQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.738739502 (512h8m50.609822548s) [noise=true , address=145.79.227.11:4732 , piece_id=K34RSS27X4CX5QMGUM4KFXO4KAD5LIXGIT3L6NPSKS5LBTJQLW2A, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.907368801 (512h8m50.778451876s) [noise=true , address=145.79.227.34:4906 , piece_id=4PUAEFIGMBKL32PDPNFNU4Y3QMQQOWJPIMDHVXBWAEKYGVNXUFYA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.929756299 (512h8m50.800839335s) [noise=true , address=45.91.116.12:4227 , piece_id=X6OFRSYOGWWHAL43G5QGHQQ6LOHO5CHJ5XUL7TRA7QNRJJQZ5CXA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.940046224 (512h8m50.811129269s) [noise=true , address=77.90.31.17:4773 , piece_id=BS5HWYDC5ULS7AZZKLV2XCWEDBQJU7V5OLDYYRLAHF7XEFO4Y7OA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.960998531 (512h8m50.832081576s) [noise=true , address=77.90.31.18:4805 , piece_id=5BKJYORH3KO7NZ6WZFKI5XYZS2N2HOQTZAWAIMXSWWPON7Z3AA3A, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:51.001412677 (512h8m50.872495732s) [noise=true , address=77.90.31.10:4519 , piece_id=IY4SR4GQG5S3FZWGLFTYDLTJNJNSSAVTXUONL32DVYQD47D4NKHQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:51.001436283 (512h8m50.872519328s) [seg_pos={0 0}, upload=449394]: Segment reached optimal threshold of 13 pieces. Dec 26 23:37:51.001662513 (512h8m50.872745558s) [noise=true , address=45.91.116.4:4940 , piece_id=VJJRUQMKNIUQ5U4TUSVISRUZFZYGBGMIE5BJG3KN5ILK6K3JCDOA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 23:37:51.001671019 (512h8m50.872754064s) [noise=true , address=45.91.116.4:4940 , piece_id=VJJRUQMKNIUQ5U4TUSVISRUZFZYGBGMIE5BJG3KN5ILK6K3JCDOA, seg_pos={0 0}, upload=449394]: Long tail cancellation due to optimal threshold Dec 26 23:37:51.001867173 (512h8m50.872950218s) [noise=true , address=145.79.227.41:4087 , piece_id=E7S2RZMLYZ3PRPT33TGFN2DCQUZ6LKS2VKZDH6RHFAJKE4OEWBTQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:57660->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:57660->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57660->145.79.227.41:4087: use of closed network connection Dec 26 23:37:51.001875027 (512h8m50.872958072s) [noise=true , address=145.79.227.41:4087 , piece_id=E7S2RZMLYZ3PRPT33TGFN2DCQUZ6LKS2VKZDH6RHFAJKE4OEWBTQ, seg_pos={0 0}, upload=449394]: Long tail cancellation due to optimal threshold Dec 26 23:37:51.002024261 (512h8m50.873107306s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKSQ5XJI5K36RS3NPCBP4E7QP4EB7MZFUJL3TA5XW5NLA3CMPXQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 23:37:51.002032206 (512h8m50.873115241s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKSQ5XJI5K36RS3NPCBP4E7QP4EB7MZFUJL3TA5XW5NLA3CMPXQ, seg_pos={0 0}, upload=449394]: Long tail cancellation due to optimal threshold Dec 26 23:37:51.002141915 (512h8m50.87322497s) [seg_pos={0 0}, upload=449394]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:37:51.002312480 (512h8m50.873395515s) [upload=449394]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:37:51.002320615 (512h8m50.87340366s) [upload=449394]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:37:51.308313376 (512h8m51.179396432s) [upload=449394]: Done uploading segments... Dec 26 23:37:51.308334487 (512h8m51.179417522s) [upload=449394]: Upload finished. err: <nil> Duration: 29.673405329s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.040430614s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:38:22 UTC) orbiter warning: context deadline exceeded Duration: 6.040468367s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............................X.X.X.X.XX....X.X.X.......X..XX] duration: 30.001801877s trace: https://jaeger.dev.storj.io/trace/5f8525aea4699e15 FAIL (2025-12-26 23:38:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: IY4SR4GQG5S3FZWGLFTYDLTJNJNSSAVTXUONL32DVYQD47D4NKHQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 5BKJYORH3KO7NZ6WZFKI5XYZS2N2HOQTZAWAIMXSWWPON7Z3AA3A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: BS5HWYDC5ULS7AZZKLV2XCWEDBQJU7V5OLDYYRLAHF7XEFO4Y7OA) context deadline exceeded ======LOGS====== Dec 26 23:37:48.433532566 (512h8m48.304615601s) [upload=449394]: Starting upload Dec 26 23:37:48.433597038 (512h8m48.304680084s) [upload=449394]: Done starting upload Dec 26 23:37:48.433763325 (512h8m48.30484636s) [upload=449394]: Uploading segments... Dec 26 23:37:48.433941715 (512h8m48.30502476s) [upload=449394]: Got next segment. Inline: false Dec 26 23:37:48.434008202 (512h8m48.305091247s) [upload=449394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:37:48.434013642 (512h8m48.305096687s) [upload=449394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:37:48.678938819 (512h8m48.550021864s) [seg_pos={0 0}, upload=449394]: Begin upload segment... Dec 26 23:37:48.678993332 (512h8m48.550076367s) [seg_pos={0 0}, upload=449394]: Done begin upload segment. Dec 26 23:37:48.679022328 (512h8m48.550105363s) [noise=true , address=77.90.31.17:4773 , piece_id=BS5HWYDC5ULS7AZZKLV2XCWEDBQJU7V5OLDYYRLAHF7XEFO4Y7OA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.679084987 (512h8m48.550168022s) [noise=true , address=77.90.31.10:4519 , piece_id=IY4SR4GQG5S3FZWGLFTYDLTJNJNSSAVTXUONL32DVYQD47D4NKHQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.679602171 (512h8m48.550685216s) [noise=true , address=145.79.227.50:4375 , piece_id=YA4O4ZBZZ7ISPINUCKF7QWPREXJGFULRGYOAJCWUHPLXZDZJ3XLQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.680053741 (512h8m48.551136786s) [noise=true , address=145.79.227.46:4603 , piece_id=NKTJ4MWPFFNAJV3PBMBVRNKEOO6PPLNEOLMG2VR2STBFTIH4BTLQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.680478409 (512h8m48.551561444s) [noise=true , address=145.79.227.45:4231 , piece_id=BITLXZF7OZOOEQQS5OD33H537XTXI4IPZO46QBREFVTC3RX4QG7A, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.680742893 (512h8m48.551825928s) [noise=true , address=77.90.31.18:4805 , piece_id=5BKJYORH3KO7NZ6WZFKI5XYZS2N2HOQTZAWAIMXSWWPON7Z3AA3A, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.681052100 (512h8m48.552135136s) [noise=true , address=45.91.116.12:4227 , piece_id=X6OFRSYOGWWHAL43G5QGHQQ6LOHO5CHJ5XUL7TRA7QNRJJQZ5CXA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.681348726 (512h8m48.552431761s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKSQ5XJI5K36RS3NPCBP4E7QP4EB7MZFUJL3TA5XW5NLA3CMPXQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.681645170 (512h8m48.552728205s) [noise=true , address=145.79.227.34:4906 , piece_id=4PUAEFIGMBKL32PDPNFNU4Y3QMQQOWJPIMDHVXBWAEKYGVNXUFYA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.683243141 (512h8m48.554326176s) [noise=true , address=145.79.227.41:4087 , piece_id=E7S2RZMLYZ3PRPT33TGFN2DCQUZ6LKS2VKZDH6RHFAJKE4OEWBTQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.684298108 (512h8m48.555381143s) [noise=true , address=145.79.227.19:4079 , piece_id=BT4BG5TZ5N5UYNXRX6L2UPXDYKX5LXRHQBYRLV7VRUR2FHBYVCNA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.685503052 (512h8m48.556586087s) [noise=true , address=145.79.227.13:4802 , piece_id=SH2YPBQOA6G6ZOQB25ZYNTNLV3TY5SFZQRUK42CJVIAPV2JMFYKA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.686990402 (512h8m48.558073438s) [noise=true , address=145.79.227.26:4653 , piece_id=R3USFFWWIEKPAIPVRGGFKUBCJVNAXHXEUTRLNO7KGQUXHKRGIVEQ, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.688191329 (512h8m48.559274374s) [noise=true , address=45.91.116.4:4940 , piece_id=VJJRUQMKNIUQ5U4TUSVISRUZFZYGBGMIE5BJG3KN5ILK6K3JCDOA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.691727599 (512h8m48.562810644s) [noise=true , address=145.79.227.48:4303 , piece_id=XIHPVIT3EYTA4SVYZ2STPWDSUK7NCR6MHWPVHPLOT2FMRUR5LUKA, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:48.693260165 (512h8m48.5643432s) [noise=true , address=145.79.227.11:4732 , piece_id=K34RSS27X4CX5QMGUM4KFXO4KAD5LIXGIT3L6NPSKS5LBTJQLW2A, seg_pos={0 0}, upload=449394]: Uploading piece... Dec 26 23:37:50.382876507 (512h8m50.253959552s) [upload=449394]: Next returned nil segment Dec 26 23:37:50.382894812 (512h8m50.253977857s) [upload=449394]: Waiting for error group managing segments... Dec 26 23:37:50.669447996 (512h8m50.540531031s) [noise=true , address=145.79.227.46:4603 , piece_id=NKTJ4MWPFFNAJV3PBMBVRNKEOO6PPLNEOLMG2VR2STBFTIH4BTLQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.671907436 (512h8m50.542990472s) [noise=true , address=145.79.227.45:4231 , piece_id=BITLXZF7OZOOEQQS5OD33H537XTXI4IPZO46QBREFVTC3RX4QG7A, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.678604112 (512h8m50.549687157s) [noise=true , address=145.79.227.50:4375 , piece_id=YA4O4ZBZZ7ISPINUCKF7QWPREXJGFULRGYOAJCWUHPLXZDZJ3XLQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.701564421 (512h8m50.572647456s) [noise=true , address=145.79.227.13:4802 , piece_id=SH2YPBQOA6G6ZOQB25ZYNTNLV3TY5SFZQRUK42CJVIAPV2JMFYKA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.703755230 (512h8m50.574838265s) [noise=true , address=145.79.227.48:4303 , piece_id=XIHPVIT3EYTA4SVYZ2STPWDSUK7NCR6MHWPVHPLOT2FMRUR5LUKA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.720142769 (512h8m50.591225804s) [noise=true , address=145.79.227.19:4079 , piece_id=BT4BG5TZ5N5UYNXRX6L2UPXDYKX5LXRHQBYRLV7VRUR2FHBYVCNA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.724516142 (512h8m50.595599177s) [noise=true , address=145.79.227.26:4653 , piece_id=R3USFFWWIEKPAIPVRGGFKUBCJVNAXHXEUTRLNO7KGQUXHKRGIVEQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.738739502 (512h8m50.609822548s) [noise=true , address=145.79.227.11:4732 , piece_id=K34RSS27X4CX5QMGUM4KFXO4KAD5LIXGIT3L6NPSKS5LBTJQLW2A, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.907368801 (512h8m50.778451876s) [noise=true , address=145.79.227.34:4906 , piece_id=4PUAEFIGMBKL32PDPNFNU4Y3QMQQOWJPIMDHVXBWAEKYGVNXUFYA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.929756299 (512h8m50.800839335s) [noise=true , address=45.91.116.12:4227 , piece_id=X6OFRSYOGWWHAL43G5QGHQQ6LOHO5CHJ5XUL7TRA7QNRJJQZ5CXA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.940046224 (512h8m50.811129269s) [noise=true , address=77.90.31.17:4773 , piece_id=BS5HWYDC5ULS7AZZKLV2XCWEDBQJU7V5OLDYYRLAHF7XEFO4Y7OA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:50.960998531 (512h8m50.832081576s) [noise=true , address=77.90.31.18:4805 , piece_id=5BKJYORH3KO7NZ6WZFKI5XYZS2N2HOQTZAWAIMXSWWPON7Z3AA3A, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:51.001412677 (512h8m50.872495732s) [noise=true , address=77.90.31.10:4519 , piece_id=IY4SR4GQG5S3FZWGLFTYDLTJNJNSSAVTXUONL32DVYQD47D4NKHQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: <nil> Dec 26 23:37:51.001436283 (512h8m50.872519328s) [seg_pos={0 0}, upload=449394]: Segment reached optimal threshold of 13 pieces. Dec 26 23:37:51.001662513 (512h8m50.872745558s) [noise=true , address=45.91.116.4:4940 , piece_id=VJJRUQMKNIUQ5U4TUSVISRUZFZYGBGMIE5BJG3KN5ILK6K3JCDOA, seg_pos={0 0}, upload=449394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 23:37:51.001671019 (512h8m50.872754064s) [noise=true , address=45.91.116.4:4940 , piece_id=VJJRUQMKNIUQ5U4TUSVISRUZFZYGBGMIE5BJG3KN5ILK6K3JCDOA, seg_pos={0 0}, upload=449394]: Long tail cancellation due to optimal threshold Dec 26 23:37:51.001867173 (512h8m50.872950218s) [noise=true , address=145.79.227.41:4087 , piece_id=E7S2RZMLYZ3PRPT33TGFN2DCQUZ6LKS2VKZDH6RHFAJKE4OEWBTQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:57660->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:57660->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57660->145.79.227.41:4087: use of closed network connection Dec 26 23:37:51.001875027 (512h8m50.872958072s) [noise=true , address=145.79.227.41:4087 , piece_id=E7S2RZMLYZ3PRPT33TGFN2DCQUZ6LKS2VKZDH6RHFAJKE4OEWBTQ, seg_pos={0 0}, upload=449394]: Long tail cancellation due to optimal threshold Dec 26 23:37:51.002024261 (512h8m50.873107306s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKSQ5XJI5K36RS3NPCBP4E7QP4EB7MZFUJL3TA5XW5NLA3CMPXQ, seg_pos={0 0}, upload=449394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 23:37:51.002032206 (512h8m50.873115241s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKSQ5XJI5K36RS3NPCBP4E7QP4EB7MZFUJL3TA5XW5NLA3CMPXQ, seg_pos={0 0}, upload=449394]: Long tail cancellation due to optimal threshold Dec 26 23:37:51.002141915 (512h8m50.87322497s) [seg_pos={0 0}, upload=449394]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:37:51.002312480 (512h8m50.873395515s) [upload=449394]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:37:51.002320615 (512h8m50.87340366s) [upload=449394]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:37:51.308313376 (512h8m51.179396432s) [upload=449394]: Done uploading segments... Dec 26 23:37:51.308334487 (512h8m51.179417522s) [upload=449394]: Upload finished. err: <nil> Duration: 30.001823559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............................X.X.X.X.XX....X.X.X.......X..X] duration: 30.003040315s trace: https://jaeger.dev.storj.io/trace/174a0e4aa5c13c89 FAIL (2025-12-26 23:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: MQMWK4CLEKSHS3CRKWESDRGJB5GVEEIEHODZFU65Y55677TZAA4Q) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: VHTIPM4KLTPYPD2WWRVODSBYBQ23AXA73DG4LAKO6D3VKUA7DNCA) context deadline exceeded ======LOGS====== Dec 26 23:37:17.147006958 (512h8m17.018089993s) [upload=449388]: Starting upload Dec 26 23:37:17.147068285 (512h8m17.01815132s) [upload=449388]: Done starting upload Dec 26 23:37:17.147181160 (512h8m17.018264215s) [upload=449388]: Uploading segments... Dec 26 23:37:17.147331195 (512h8m17.01841423s) [upload=449388]: Got next segment. Inline: false Dec 26 23:37:17.147395849 (512h8m17.018478894s) [upload=449388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:37:17.147402641 (512h8m17.018485686s) [upload=449388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:37:17.393761276 (512h8m17.264844321s) [seg_pos={0 0}, upload=449388]: Begin upload segment... Dec 26 23:37:17.393862519 (512h8m17.264945554s) [seg_pos={0 0}, upload=449388]: Done begin upload segment. Dec 26 23:37:17.393890773 (512h8m17.264973818s) [noise=true , address=77.90.31.16:4735 , piece_id=J4FOX377HEE47LN6OOREZ4CCPIDPBAPTRUYC5DIVCWK7VBDE3EPQ, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.393928134 (512h8m17.265011169s) [noise=true , address=77.90.31.10:4518 , piece_id=7SL4VVSQZHHR4FZJEATE4NH37JVFCKFKWSJDRNMNQYUJNMHKW47Q, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.394620391 (512h8m17.265703426s) [noise=true , address=45.91.116.6:4975 , piece_id=LBRJIG3DIVLEEAW2ORHPHD32P3CHOOMGLMWO3JJYMRK4WQVSJKMQ, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.395169928 (512h8m17.266252963s) [noise=true , address=145.79.227.11:4729 , piece_id=VFMGLSSZZQPXLYDAYWSU7SQFQY4EPVEJJL2CYD3GRNCNJM6B7ICA, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.395592101 (512h8m17.266675136s) [noise=true , address=45.91.116.7:4011 , piece_id=VHTIPM4KLTPYPD2WWRVODSBYBQ23AXA73DG4LAKO6D3VKUA7DNCA, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.395981712 (512h8m17.267064747s) [noise=true , address=45.91.116.5:4300 , piece_id=ZYZLAAPKXOVFH64DN5LJQ4J2ESL3SMACWVVRIMIXLHW6EOEZ7HCQ, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.396377696 (512h8m17.267460741s) [noise=true , address=145.79.227.42:4123 , piece_id=YMD4BB3PM4RW5J4XMHYGAR2DW2JREAIWRDWFPU3YGUOWCHBUUB4A, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.397890906 (512h8m17.268973971s) [noise=true , address=145.79.227.41:4086 , piece_id=NKB5RSL74W6KA2KZJIVYAPD263KIBTZXDUM3KGCMTJDD2VF7QRMQ, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.398763396 (512h8m17.269846461s) [noise=true , address=145.79.227.48:4303 , piece_id=2NGHTTUP3ND3LCEUUODFIAFDBZ6DQFCIXGKWUWVSM3VFI7ACPI2Q, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.399411931 (512h8m17.270494976s) [noise=true , address=145.79.227.46:4603 , piece_id=M4LQZXMSY4MOSLCHHKLOPRZLR72R6EWERKXV7PJCIYS4EQ6VYGQQ, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.399952089 (512h8m17.271035124s) [noise=true , address=145.79.227.45:4230 , piece_id=TZ45J4WAU7T4VW3L2S54OOKGAS7K3DI3UATDYM3VYYOLQCPRYRKA, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.400398399 (512h8m17.271481434s) [noise=true , address=45.91.116.9:4261 , piece_id=MQMWK4CLEKSHS3CRKWESDRGJB5GVEEIEHODZFU65Y55677TZAA4Q, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.401910216 (512h8m17.272993251s) [noise=true , address=145.79.227.23:4545 , piece_id=4EXZJGLGBK3UFZJ3PVSZCABI2J2UYXQRFLQT2HGHKMP3P5RQXXOA, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.403268010 (512h8m17.274351045s) [noise=true , address=77.90.31.19:4844 , piece_id=E6FQTHPTSVXBLYLE2RCCNSHWMKGIGTU3ISEJX2UTO5BVO6UR5JRQ, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.404660240 (512h8m17.275743285s) [noise=true , address=145.79.227.15:4876 , piece_id=PVEYCFG3WXVIVZYOPW2XR5KHECZBFU3M63ABP26EBDY7U7EII7EQ, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:17.406222133 (512h8m17.277305168s) [noise=true , address=77.90.31.17:4773 , piece_id=TCVAMNR7WGBDBHYKJQEB36GIDRKLHGDD6GT62XLPXKLY4LZ33XTA, seg_pos={0 0}, upload=449388]: Uploading piece... Dec 26 23:37:19.003123756 (512h8m18.874206801s) [upload=449388]: Next returned nil segment Dec 26 23:37:19.003135749 (512h8m18.874218794s) [upload=449388]: Waiting for error group managing segments... Dec 26 23:37:19.387830147 (512h8m19.258913182s) [noise=true , address=145.79.227.42:4123 , piece_id=YMD4BB3PM4RW5J4XMHYGAR2DW2JREAIWRDWFPU3YGUOWCHBUUB4A, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.392049899 (512h8m19.263132934s) [noise=true , address=145.79.227.46:4603 , piece_id=M4LQZXMSY4MOSLCHHKLOPRZLR72R6EWERKXV7PJCIYS4EQ6VYGQQ, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.402639142 (512h8m19.273722177s) [noise=true , address=145.79.227.45:4230 , piece_id=TZ45J4WAU7T4VW3L2S54OOKGAS7K3DI3UATDYM3VYYOLQCPRYRKA, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.408822340 (512h8m19.279905375s) [noise=true , address=145.79.227.48:4303 , piece_id=2NGHTTUP3ND3LCEUUODFIAFDBZ6DQFCIXGKWUWVSM3VFI7ACPI2Q, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.412492585 (512h8m19.28357562s) [noise=true , address=145.79.227.15:4876 , piece_id=PVEYCFG3WXVIVZYOPW2XR5KHECZBFU3M63ABP26EBDY7U7EII7EQ, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.414293192 (512h8m19.285376227s) [noise=true , address=145.79.227.11:4729 , piece_id=VFMGLSSZZQPXLYDAYWSU7SQFQY4EPVEJJL2CYD3GRNCNJM6B7ICA, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.431802005 (512h8m19.30288508s) [noise=true , address=145.79.227.41:4086 , piece_id=NKB5RSL74W6KA2KZJIVYAPD263KIBTZXDUM3KGCMTJDD2VF7QRMQ, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.637926806 (512h8m19.509009841s) [noise=true , address=45.91.116.7:4011 , piece_id=VHTIPM4KLTPYPD2WWRVODSBYBQ23AXA73DG4LAKO6D3VKUA7DNCA, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.650850132 (512h8m19.521933177s) [noise=true , address=45.91.116.6:4975 , piece_id=LBRJIG3DIVLEEAW2ORHPHD32P3CHOOMGLMWO3JJYMRK4WQVSJKMQ, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.655631452 (512h8m19.526714487s) [noise=true , address=77.90.31.19:4844 , piece_id=E6FQTHPTSVXBLYLE2RCCNSHWMKGIGTU3ISEJX2UTO5BVO6UR5JRQ, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.673314677 (512h8m19.544397712s) [noise=true , address=45.91.116.9:4261 , piece_id=MQMWK4CLEKSHS3CRKWESDRGJB5GVEEIEHODZFU65Y55677TZAA4Q, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.740514132 (512h8m19.611597167s) [noise=true , address=77.90.31.17:4773 , piece_id=TCVAMNR7WGBDBHYKJQEB36GIDRKLHGDD6GT62XLPXKLY4LZ33XTA, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.969841062 (512h8m19.840924097s) [noise=true , address=45.91.116.5:4300 , piece_id=ZYZLAAPKXOVFH64DN5LJQ4J2ESL3SMACWVVRIMIXLHW6EOEZ7HCQ, seg_pos={0 0}, upload=449388]: Done uploading piece. err: <nil> Dec 26 23:37:19.969859987 (512h8m19.840943022s) [seg_pos={0 0}, upload=449388]: Segment reached optimal threshold of 13 pieces. Dec 26 23:37:19.970106677 (512h8m19.841189712s) [noise=true , address=77.90.31.10:4518 , piece_id=7SL4VVSQZHHR4FZJEATE4NH37JVFCKFKWSJDRNMNQYUJNMHKW47Q, seg_pos={0 0}, upload=449388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 23:37:19.970114572 (512h8m19.841197607s) [noise=true , address=77.90.31.10:4518 , piece_id=7SL4VVSQZHHR4FZJEATE4NH37JVFCKFKWSJDRNMNQYUJNMHKW47Q, seg_pos={0 0}, upload=449388]: Long tail cancellation due to optimal threshold Dec 26 23:37:19.970464959 (512h8m19.841548004s) [noise=true , address=77.90.31.16:4735 , piece_id=J4FOX377HEE47LN6OOREZ4CCPIDPBAPTRUYC5DIVCWK7VBDE3EPQ, seg_pos={0 0}, upload=449388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 23:37:19.970475148 (512h8m19.841558193s) [noise=true , address=77.90.31.16:4735 , piece_id=J4FOX377HEE47LN6OOREZ4CCPIDPBAPTRUYC5DIVCWK7VBDE3EPQ, seg_pos={0 0}, upload=449388]: Long tail cancellation due to optimal threshold Dec 26 23:37:19.970669899 (512h8m19.841752934s) [noise=true , address=145.79.227.23:4545 , piece_id=4EXZJGLGBK3UFZJ3PVSZCABI2J2UYXQRFLQT2HGHKMP3P5RQXXOA, seg_pos={0 0}, upload=449388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:60570->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:60570->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60570->145.79.227.23:4545: use of closed network connection Dec 26 23:37:19.970676712 (512h8m19.841759757s) [noise=true , address=145.79.227.23:4545 , piece_id=4EXZJGLGBK3UFZJ3PVSZCABI2J2UYXQRFLQT2HGHKMP3P5RQXXOA, seg_pos={0 0}, upload=449388]: Long tail cancellation due to optimal threshold Dec 26 23:37:19.970693655 (512h8m19.84177669s) [seg_pos={0 0}, upload=449388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:37:19.970768137 (512h8m19.841851172s) [upload=449388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:37:19.970772415 (512h8m19.84185546s) [upload=449388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:37:20.268574327 (512h8m20.139657372s) [upload=449388]: Done uploading segments... Dec 26 23:37:20.268598452 (512h8m20.139681487s) [upload=449388]: Upload finished. err: <nil> Duration: 30.003051637s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.285886318s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:37:08 UTC) orbiter warning: context deadline exceeded Duration: 6.285915424s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..............................X.X.X.X.XX....X.X.X.......X..] duration: 11.806723192s trace: https://jaeger.dev.storj.io/trace/56c65e1e7026c58 FAIL (2025-12-26 23:36:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA) context deadline exceeded ======LOGS====== Dec 26 23:35:05.099088051 (512h6m4.970171096s) [upload=449356]: Starting upload Dec 26 23:35:05.099178734 (512h6m4.970261779s) [upload=449356]: Done starting upload Dec 26 23:35:05.099422928 (512h6m4.970505963s) [upload=449356]: Uploading segments... Dec 26 23:35:05.099472343 (512h6m4.970555378s) [upload=449356]: Got next segment. Inline: false Dec 26 23:35:05.099508161 (512h6m4.970591206s) [upload=449356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:35:05.099514012 (512h6m4.970597057s) [upload=449356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:35:05.342695089 (512h6m5.213778134s) [seg_pos={0 0}, upload=449356]: Begin upload segment... Dec 26 23:35:05.342766305 (512h6m5.21384935s) [seg_pos={0 0}, upload=449356]: Done begin upload segment. Dec 26 23:35:05.342804537 (512h6m5.213887572s) [noise=true , address=145.79.227.3:4151 , piece_id=HR33RMTU2EZDTOSKIL6DFRYRL2DTKYA7B4CIU44BY7FOB3KSMQZQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.342940927 (512h6m5.214023972s) [noise=true , address=145.79.227.45:4231 , piece_id=SZHFZJXXTYQWLFYWHBUGTTMCS4KSMTOQV5V25M6MWTS4M6CGQBYA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.343547261 (512h6m5.214630306s) [noise=true , address=45.91.116.9:4261 , piece_id=RTBGH2CVN7JEVSJZLEHC4VOZ4SHLSW2ZA4EC66WR7G4HJRISZWSA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.343976718 (512h6m5.215059763s) [noise=true , address=145.79.227.36:4517 , piece_id=OYTRFSV7HQQ3CASDEGUC47POEVZ766S6AGK2QEGN73VRFMPLH4LA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344321114 (512h6m5.215404149s) [noise=true , address=45.91.116.12:4228 , piece_id=RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344765790 (512h6m5.215848825s) [noise=true , address=145.79.227.37:4567 , piece_id=CQJTJSZW5HBXJUUN2OZDE3DGBAVZH6MYMQXPITKNVW3WYG2IUUQQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344985728 (512h6m5.216068773s) [noise=true , address=145.79.227.11:4729 , piece_id=TFLAWZOKWHB3YCRZXJEMFWIH4KEIRJOUDCTLI73TZKAGKHRXOHWQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345290338 (512h6m5.216373373s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345513773 (512h6m5.216596818s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345765803 (512h6m5.216848838s) [noise=true , address=145.79.227.16:4909 , piece_id=CC46W6FMVUYMKWP74WQBVIYCEROPW6K5CQ3LFM6XP247C3LQPCJA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345884308 (512h6m5.216967353s) [noise=true , address=145.79.227.13:4802 , piece_id=A2EA6SELWDYJVAU67GGIX3URLTJFQ4NXQSRFGY5BS2RP4N6DA47A, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346374532 (512h6m5.217457577s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346514869 (512h6m5.217597914s) [noise=true , address=145.79.227.22:4510 , piece_id=E2K6JEVCV6RT3G5UWARLTYVPHMNZAL7WUXQTSZBDQ3B2DU5IDLGQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346883701 (512h6m5.217966776s) [noise=true , address=77.90.31.19:4844 , piece_id=FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.347458845 (512h6m5.21854188s) [noise=true , address=77.90.31.11:4554 , piece_id=SIPUEFCKKXO7NVEV6MT23DMLBEECF2IMMF5OSLSMTCPSDGW74IXQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.347898532 (512h6m5.218981587s) [noise=true , address=145.79.227.46:4602 , piece_id=CZADJ4RL3G5CYJYPIR26L2EYDEQZIZE7IY5TUFE5ER4OQTJYZO6A, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:07.059633239 (512h6m6.930716304s) [upload=449356]: Next returned nil segment Dec 26 23:35:07.059652655 (512h6m6.9307357s) [upload=449356]: Waiting for error group managing segments... Dec 26 23:35:07.333247686 (512h6m7.204330731s) [noise=true , address=145.79.227.11:4729 , piece_id=TFLAWZOKWHB3YCRZXJEMFWIH4KEIRJOUDCTLI73TZKAGKHRXOHWQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.344578652 (512h6m7.215661697s) [noise=true , address=145.79.227.37:4567 , piece_id=CQJTJSZW5HBXJUUN2OZDE3DGBAVZH6MYMQXPITKNVW3WYG2IUUQQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.347834179 (512h6m7.218917214s) [noise=true , address=145.79.227.3:4151 , piece_id=HR33RMTU2EZDTOSKIL6DFRYRL2DTKYA7B4CIU44BY7FOB3KSMQZQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.350978703 (512h6m7.222061738s) [noise=true , address=145.79.227.13:4802 , piece_id=A2EA6SELWDYJVAU67GGIX3URLTJFQ4NXQSRFGY5BS2RP4N6DA47A, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.360502839 (512h6m7.231585874s) [noise=true , address=145.79.227.46:4602 , piece_id=CZADJ4RL3G5CYJYPIR26L2EYDEQZIZE7IY5TUFE5ER4OQTJYZO6A, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.365543322 (512h6m7.236626357s) [noise=true , address=145.79.227.36:4517 , piece_id=OYTRFSV7HQQ3CASDEGUC47POEVZ766S6AGK2QEGN73VRFMPLH4LA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.367214252 (512h6m7.238297297s) [noise=true , address=145.79.227.45:4231 , piece_id=SZHFZJXXTYQWLFYWHBUGTTMCS4KSMTOQV5V25M6MWTS4M6CGQBYA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.377902484 (512h6m7.248985519s) [noise=true , address=145.79.227.22:4510 , piece_id=E2K6JEVCV6RT3G5UWARLTYVPHMNZAL7WUXQTSZBDQ3B2DU5IDLGQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.447265087 (512h6m7.318348132s) [noise=true , address=145.79.227.16:4909 , piece_id=CC46W6FMVUYMKWP74WQBVIYCEROPW6K5CQ3LFM6XP247C3LQPCJA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.528121334 (512h6m7.399204379s) [noise=true , address=77.90.31.19:4844 , piece_id=FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.600255712 (512h6m7.471338757s) [noise=true , address=77.90.31.11:4554 , piece_id=SIPUEFCKKXO7NVEV6MT23DMLBEECF2IMMF5OSLSMTCPSDGW74IXQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.615986261 (512h6m7.487069296s) [noise=true , address=45.91.116.12:4228 , piece_id=RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.626467900 (512h6m7.497550945s) [noise=true , address=45.91.116.9:4261 , piece_id=RTBGH2CVN7JEVSJZLEHC4VOZ4SHLSW2ZA4EC66WR7G4HJRISZWSA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.626491053 (512h6m7.497574088s) [seg_pos={0 0}, upload=449356]: Segment reached optimal threshold of 13 pieces. Dec 26 23:35:07.626681837 (512h6m7.497764882s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 23:35:07.626690673 (512h6m7.497773708s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.626873641 (512h6m7.497956687s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 23:35:07.626882077 (512h6m7.497965132s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.627065817 (512h6m7.498148862s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection Dec 26 23:35:07.627073783 (512h6m7.498156838s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.627095554 (512h6m7.498178599s) [seg_pos={0 0}, upload=449356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:35:07.627170687 (512h6m7.498253732s) [upload=449356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:35:07.627176728 (512h6m7.498259773s) [upload=449356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:35:07.920152875 (512h6m7.79123591s) [upload=449356]: Done uploading segments... Dec 26 23:35:07.920172663 (512h6m7.791255698s) [upload=449356]: Upload finished. err: <nil> Duration: 11.806757647s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.38468139s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:36:09 UTC) orbiter warning: context deadline exceeded Duration: 6.384707349s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..............................X.X.X.X.XX....X.X.X.......X.] duration: 5.997261291s trace: https://jaeger.dev.storj.io/trace/56c65e1e7026c58 FAIL (2025-12-26 23:36:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA) context deadline exceeded ======LOGS====== Dec 26 23:35:05.099088051 (512h6m4.970171096s) [upload=449356]: Starting upload Dec 26 23:35:05.099178734 (512h6m4.970261779s) [upload=449356]: Done starting upload Dec 26 23:35:05.099422928 (512h6m4.970505963s) [upload=449356]: Uploading segments... Dec 26 23:35:05.099472343 (512h6m4.970555378s) [upload=449356]: Got next segment. Inline: false Dec 26 23:35:05.099508161 (512h6m4.970591206s) [upload=449356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:35:05.099514012 (512h6m4.970597057s) [upload=449356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:35:05.342695089 (512h6m5.213778134s) [seg_pos={0 0}, upload=449356]: Begin upload segment... Dec 26 23:35:05.342766305 (512h6m5.21384935s) [seg_pos={0 0}, upload=449356]: Done begin upload segment. Dec 26 23:35:05.342804537 (512h6m5.213887572s) [noise=true , address=145.79.227.3:4151 , piece_id=HR33RMTU2EZDTOSKIL6DFRYRL2DTKYA7B4CIU44BY7FOB3KSMQZQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.342940927 (512h6m5.214023972s) [noise=true , address=145.79.227.45:4231 , piece_id=SZHFZJXXTYQWLFYWHBUGTTMCS4KSMTOQV5V25M6MWTS4M6CGQBYA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.343547261 (512h6m5.214630306s) [noise=true , address=45.91.116.9:4261 , piece_id=RTBGH2CVN7JEVSJZLEHC4VOZ4SHLSW2ZA4EC66WR7G4HJRISZWSA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.343976718 (512h6m5.215059763s) [noise=true , address=145.79.227.36:4517 , piece_id=OYTRFSV7HQQ3CASDEGUC47POEVZ766S6AGK2QEGN73VRFMPLH4LA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344321114 (512h6m5.215404149s) [noise=true , address=45.91.116.12:4228 , piece_id=RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344765790 (512h6m5.215848825s) [noise=true , address=145.79.227.37:4567 , piece_id=CQJTJSZW5HBXJUUN2OZDE3DGBAVZH6MYMQXPITKNVW3WYG2IUUQQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344985728 (512h6m5.216068773s) [noise=true , address=145.79.227.11:4729 , piece_id=TFLAWZOKWHB3YCRZXJEMFWIH4KEIRJOUDCTLI73TZKAGKHRXOHWQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345290338 (512h6m5.216373373s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345513773 (512h6m5.216596818s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345765803 (512h6m5.216848838s) [noise=true , address=145.79.227.16:4909 , piece_id=CC46W6FMVUYMKWP74WQBVIYCEROPW6K5CQ3LFM6XP247C3LQPCJA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345884308 (512h6m5.216967353s) [noise=true , address=145.79.227.13:4802 , piece_id=A2EA6SELWDYJVAU67GGIX3URLTJFQ4NXQSRFGY5BS2RP4N6DA47A, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346374532 (512h6m5.217457577s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346514869 (512h6m5.217597914s) [noise=true , address=145.79.227.22:4510 , piece_id=E2K6JEVCV6RT3G5UWARLTYVPHMNZAL7WUXQTSZBDQ3B2DU5IDLGQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346883701 (512h6m5.217966776s) [noise=true , address=77.90.31.19:4844 , piece_id=FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.347458845 (512h6m5.21854188s) [noise=true , address=77.90.31.11:4554 , piece_id=SIPUEFCKKXO7NVEV6MT23DMLBEECF2IMMF5OSLSMTCPSDGW74IXQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.347898532 (512h6m5.218981587s) [noise=true , address=145.79.227.46:4602 , piece_id=CZADJ4RL3G5CYJYPIR26L2EYDEQZIZE7IY5TUFE5ER4OQTJYZO6A, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:07.059633239 (512h6m6.930716304s) [upload=449356]: Next returned nil segment Dec 26 23:35:07.059652655 (512h6m6.9307357s) [upload=449356]: Waiting for error group managing segments... Dec 26 23:35:07.333247686 (512h6m7.204330731s) [noise=true , address=145.79.227.11:4729 , piece_id=TFLAWZOKWHB3YCRZXJEMFWIH4KEIRJOUDCTLI73TZKAGKHRXOHWQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.344578652 (512h6m7.215661697s) [noise=true , address=145.79.227.37:4567 , piece_id=CQJTJSZW5HBXJUUN2OZDE3DGBAVZH6MYMQXPITKNVW3WYG2IUUQQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.347834179 (512h6m7.218917214s) [noise=true , address=145.79.227.3:4151 , piece_id=HR33RMTU2EZDTOSKIL6DFRYRL2DTKYA7B4CIU44BY7FOB3KSMQZQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.350978703 (512h6m7.222061738s) [noise=true , address=145.79.227.13:4802 , piece_id=A2EA6SELWDYJVAU67GGIX3URLTJFQ4NXQSRFGY5BS2RP4N6DA47A, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.360502839 (512h6m7.231585874s) [noise=true , address=145.79.227.46:4602 , piece_id=CZADJ4RL3G5CYJYPIR26L2EYDEQZIZE7IY5TUFE5ER4OQTJYZO6A, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.365543322 (512h6m7.236626357s) [noise=true , address=145.79.227.36:4517 , piece_id=OYTRFSV7HQQ3CASDEGUC47POEVZ766S6AGK2QEGN73VRFMPLH4LA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.367214252 (512h6m7.238297297s) [noise=true , address=145.79.227.45:4231 , piece_id=SZHFZJXXTYQWLFYWHBUGTTMCS4KSMTOQV5V25M6MWTS4M6CGQBYA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.377902484 (512h6m7.248985519s) [noise=true , address=145.79.227.22:4510 , piece_id=E2K6JEVCV6RT3G5UWARLTYVPHMNZAL7WUXQTSZBDQ3B2DU5IDLGQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.447265087 (512h6m7.318348132s) [noise=true , address=145.79.227.16:4909 , piece_id=CC46W6FMVUYMKWP74WQBVIYCEROPW6K5CQ3LFM6XP247C3LQPCJA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.528121334 (512h6m7.399204379s) [noise=true , address=77.90.31.19:4844 , piece_id=FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.600255712 (512h6m7.471338757s) [noise=true , address=77.90.31.11:4554 , piece_id=SIPUEFCKKXO7NVEV6MT23DMLBEECF2IMMF5OSLSMTCPSDGW74IXQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.615986261 (512h6m7.487069296s) [noise=true , address=45.91.116.12:4228 , piece_id=RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.626467900 (512h6m7.497550945s) [noise=true , address=45.91.116.9:4261 , piece_id=RTBGH2CVN7JEVSJZLEHC4VOZ4SHLSW2ZA4EC66WR7G4HJRISZWSA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.626491053 (512h6m7.497574088s) [seg_pos={0 0}, upload=449356]: Segment reached optimal threshold of 13 pieces. Dec 26 23:35:07.626681837 (512h6m7.497764882s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 23:35:07.626690673 (512h6m7.497773708s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.626873641 (512h6m7.497956687s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 23:35:07.626882077 (512h6m7.497965132s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.627065817 (512h6m7.498148862s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection Dec 26 23:35:07.627073783 (512h6m7.498156838s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.627095554 (512h6m7.498178599s) [seg_pos={0 0}, upload=449356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:35:07.627170687 (512h6m7.498253732s) [upload=449356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:35:07.627176728 (512h6m7.498259773s) [upload=449356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:35:07.920152875 (512h6m7.79123591s) [upload=449356]: Done uploading segments... Dec 26 23:35:07.920172663 (512h6m7.791255698s) [upload=449356]: Upload finished. err: <nil> Duration: 5.99729153s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.91485697s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:35:42 UTC) orbiter warning: context deadline exceeded Duration: 5.914882429s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..............................X.X.X.X.XX....X.X.X.......X] duration: 30.002656853s trace: https://jaeger.dev.storj.io/trace/56c65e1e7026c58 FAIL (2025-12-26 23:35:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA) context deadline exceeded ======LOGS====== Dec 26 23:35:05.099088051 (512h6m4.970171096s) [upload=449356]: Starting upload Dec 26 23:35:05.099178734 (512h6m4.970261779s) [upload=449356]: Done starting upload Dec 26 23:35:05.099422928 (512h6m4.970505963s) [upload=449356]: Uploading segments... Dec 26 23:35:05.099472343 (512h6m4.970555378s) [upload=449356]: Got next segment. Inline: false Dec 26 23:35:05.099508161 (512h6m4.970591206s) [upload=449356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:35:05.099514012 (512h6m4.970597057s) [upload=449356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:35:05.342695089 (512h6m5.213778134s) [seg_pos={0 0}, upload=449356]: Begin upload segment... Dec 26 23:35:05.342766305 (512h6m5.21384935s) [seg_pos={0 0}, upload=449356]: Done begin upload segment. Dec 26 23:35:05.342804537 (512h6m5.213887572s) [noise=true , address=145.79.227.3:4151 , piece_id=HR33RMTU2EZDTOSKIL6DFRYRL2DTKYA7B4CIU44BY7FOB3KSMQZQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.342940927 (512h6m5.214023972s) [noise=true , address=145.79.227.45:4231 , piece_id=SZHFZJXXTYQWLFYWHBUGTTMCS4KSMTOQV5V25M6MWTS4M6CGQBYA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.343547261 (512h6m5.214630306s) [noise=true , address=45.91.116.9:4261 , piece_id=RTBGH2CVN7JEVSJZLEHC4VOZ4SHLSW2ZA4EC66WR7G4HJRISZWSA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.343976718 (512h6m5.215059763s) [noise=true , address=145.79.227.36:4517 , piece_id=OYTRFSV7HQQ3CASDEGUC47POEVZ766S6AGK2QEGN73VRFMPLH4LA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344321114 (512h6m5.215404149s) [noise=true , address=45.91.116.12:4228 , piece_id=RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344765790 (512h6m5.215848825s) [noise=true , address=145.79.227.37:4567 , piece_id=CQJTJSZW5HBXJUUN2OZDE3DGBAVZH6MYMQXPITKNVW3WYG2IUUQQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.344985728 (512h6m5.216068773s) [noise=true , address=145.79.227.11:4729 , piece_id=TFLAWZOKWHB3YCRZXJEMFWIH4KEIRJOUDCTLI73TZKAGKHRXOHWQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345290338 (512h6m5.216373373s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345513773 (512h6m5.216596818s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345765803 (512h6m5.216848838s) [noise=true , address=145.79.227.16:4909 , piece_id=CC46W6FMVUYMKWP74WQBVIYCEROPW6K5CQ3LFM6XP247C3LQPCJA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.345884308 (512h6m5.216967353s) [noise=true , address=145.79.227.13:4802 , piece_id=A2EA6SELWDYJVAU67GGIX3URLTJFQ4NXQSRFGY5BS2RP4N6DA47A, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346374532 (512h6m5.217457577s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346514869 (512h6m5.217597914s) [noise=true , address=145.79.227.22:4510 , piece_id=E2K6JEVCV6RT3G5UWARLTYVPHMNZAL7WUXQTSZBDQ3B2DU5IDLGQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.346883701 (512h6m5.217966776s) [noise=true , address=77.90.31.19:4844 , piece_id=FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.347458845 (512h6m5.21854188s) [noise=true , address=77.90.31.11:4554 , piece_id=SIPUEFCKKXO7NVEV6MT23DMLBEECF2IMMF5OSLSMTCPSDGW74IXQ, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:05.347898532 (512h6m5.218981587s) [noise=true , address=145.79.227.46:4602 , piece_id=CZADJ4RL3G5CYJYPIR26L2EYDEQZIZE7IY5TUFE5ER4OQTJYZO6A, seg_pos={0 0}, upload=449356]: Uploading piece... Dec 26 23:35:07.059633239 (512h6m6.930716304s) [upload=449356]: Next returned nil segment Dec 26 23:35:07.059652655 (512h6m6.9307357s) [upload=449356]: Waiting for error group managing segments... Dec 26 23:35:07.333247686 (512h6m7.204330731s) [noise=true , address=145.79.227.11:4729 , piece_id=TFLAWZOKWHB3YCRZXJEMFWIH4KEIRJOUDCTLI73TZKAGKHRXOHWQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.344578652 (512h6m7.215661697s) [noise=true , address=145.79.227.37:4567 , piece_id=CQJTJSZW5HBXJUUN2OZDE3DGBAVZH6MYMQXPITKNVW3WYG2IUUQQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.347834179 (512h6m7.218917214s) [noise=true , address=145.79.227.3:4151 , piece_id=HR33RMTU2EZDTOSKIL6DFRYRL2DTKYA7B4CIU44BY7FOB3KSMQZQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.350978703 (512h6m7.222061738s) [noise=true , address=145.79.227.13:4802 , piece_id=A2EA6SELWDYJVAU67GGIX3URLTJFQ4NXQSRFGY5BS2RP4N6DA47A, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.360502839 (512h6m7.231585874s) [noise=true , address=145.79.227.46:4602 , piece_id=CZADJ4RL3G5CYJYPIR26L2EYDEQZIZE7IY5TUFE5ER4OQTJYZO6A, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.365543322 (512h6m7.236626357s) [noise=true , address=145.79.227.36:4517 , piece_id=OYTRFSV7HQQ3CASDEGUC47POEVZ766S6AGK2QEGN73VRFMPLH4LA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.367214252 (512h6m7.238297297s) [noise=true , address=145.79.227.45:4231 , piece_id=SZHFZJXXTYQWLFYWHBUGTTMCS4KSMTOQV5V25M6MWTS4M6CGQBYA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.377902484 (512h6m7.248985519s) [noise=true , address=145.79.227.22:4510 , piece_id=E2K6JEVCV6RT3G5UWARLTYVPHMNZAL7WUXQTSZBDQ3B2DU5IDLGQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.447265087 (512h6m7.318348132s) [noise=true , address=145.79.227.16:4909 , piece_id=CC46W6FMVUYMKWP74WQBVIYCEROPW6K5CQ3LFM6XP247C3LQPCJA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.528121334 (512h6m7.399204379s) [noise=true , address=77.90.31.19:4844 , piece_id=FYKCCPKHTDS7MQLLM2M4H6A5RMNZG3QG2GQXWJEFWSM6JHTMBBGA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.600255712 (512h6m7.471338757s) [noise=true , address=77.90.31.11:4554 , piece_id=SIPUEFCKKXO7NVEV6MT23DMLBEECF2IMMF5OSLSMTCPSDGW74IXQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.615986261 (512h6m7.487069296s) [noise=true , address=45.91.116.12:4228 , piece_id=RGSRST6NOUMQNVNVWPOJJ5GWBLM54X3HLDFOGN2QKOLKPNZ6XF2Q, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.626467900 (512h6m7.497550945s) [noise=true , address=45.91.116.9:4261 , piece_id=RTBGH2CVN7JEVSJZLEHC4VOZ4SHLSW2ZA4EC66WR7G4HJRISZWSA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: <nil> Dec 26 23:35:07.626491053 (512h6m7.497574088s) [seg_pos={0 0}, upload=449356]: Segment reached optimal threshold of 13 pieces. Dec 26 23:35:07.626681837 (512h6m7.497764882s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 23:35:07.626690673 (512h6m7.497773708s) [noise=true , address=45.91.116.11:4119 , piece_id=BFU3ZPSLYVDLA3W6KBZVNAIYH6X3KI464M6ERKKAKOC5PLVPWFKQ, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.626873641 (512h6m7.497956687s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 23:35:07.626882077 (512h6m7.497965132s) [noise=true , address=45.91.116.5:4297 , piece_id=NU4CB5VHI4YJN23VWRELP3N5W5AR3O6EPYYOJOT7CDJXFL6TVPPA, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.627065817 (512h6m7.498148862s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37462->45.91.116.10:4082: use of closed network connection Dec 26 23:35:07.627073783 (512h6m7.498156838s) [noise=true , address=45.91.116.10:4082 , piece_id=2XT2GEOOYPVHFPMGWCB55TWSG5VRPEXT6WGN3XJYEQNMFJCUVYBA, seg_pos={0 0}, upload=449356]: Long tail cancellation due to optimal threshold Dec 26 23:35:07.627095554 (512h6m7.498178599s) [seg_pos={0 0}, upload=449356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:35:07.627170687 (512h6m7.498253732s) [upload=449356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:35:07.627176728 (512h6m7.498259773s) [upload=449356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:35:07.920152875 (512h6m7.79123591s) [upload=449356]: Done uploading segments... Dec 26 23:35:07.920172663 (512h6m7.791255698s) [upload=449356]: Upload finished. err: <nil> Duration: 30.002682993s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.845673439s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:34:35 UTC) orbiter warning: context deadline exceeded Duration: 5.845701081s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..............................X.X.X.X.XX....X.X.X.......] duration: 6.043519523s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:34:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 6.043545413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X..............................X.X.X.X.XX....X.X.X......] duration: 19.025141668s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:34:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 19.02516953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X..............................X.X.X.X.XX....X.X.X.....] duration: 6.077306014s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:33:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 6.077333428s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.072848503s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:33:34 UTC) orbiter warning: context deadline exceeded Duration: 6.072887577s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X..............................X.X.X.X.XX....X.X.X....] duration: 6.203506944s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:33:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 6.203540947s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.234916874s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:33:01 UTC) orbiter warning: context deadline exceeded Duration: 6.234960716s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.855234346s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:32:09 UTC) orbiter warning: context deadline exceeded Duration: 5.855262108s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X..............................X.X.X.X.XX....X.X.X...] duration: 6.229973993s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:32:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 6.230001465s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.893664463s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:31:45 UTC) orbiter warning: context deadline exceeded Duration: 5.893685783s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.259567751s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:31:11 UTC) orbiter warning: context deadline exceeded Duration: 6.259597899s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.......X..............................X.X.X.X.XX....X.X.X..] duration: 6.00225409s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:31:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 6.002281382s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.832494214s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:30:35 UTC) orbiter warning: context deadline exceeded Duration: 5.832532227s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.......X..............................X.X.X.X.XX....X.X.X.] duration: 8.054581582s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:30:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 8.054611598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.......X..............................X.X.X.X.XX....X.X.X] duration: 30.002389499s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 30.002410478s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.217842924s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:29:27 UTC) orbiter warning: context deadline exceeded Duration: 6.217870805s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.......X..............................X.X.X.X.XX....X.X.] duration: 16.343954836s trace: https://jaeger.dev.storj.io/trace/fa7a2ac26cbf8ad FAIL (2025-12-26 23:28:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A) context deadline exceeded ======LOGS====== Dec 26 23:27:51.598399489 (511h58m51.469482524s) [upload=449251]: Starting upload Dec 26 23:27:51.598502145 (511h58m51.46958521s) [upload=449251]: Done starting upload Dec 26 23:27:51.598595983 (511h58m51.469679038s) [upload=449251]: Uploading segments... Dec 26 23:27:51.598696234 (511h58m51.469779279s) [upload=449251]: Got next segment. Inline: false Dec 26 23:27:51.598737111 (511h58m51.469820157s) [upload=449251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:27:51.598743323 (511h58m51.469826378s) [upload=449251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:27:51.841567824 (511h58m51.712650869s) [seg_pos={0 0}, upload=449251]: Begin upload segment... Dec 26 23:27:51.841616236 (511h58m51.712699271s) [seg_pos={0 0}, upload=449251]: Done begin upload segment. Dec 26 23:27:51.841641194 (511h58m51.712724229s) [noise=true , address=77.90.31.19:4842 , piece_id=5J7XL7STKUPXR4UTUGYN24RBQUAQW4QWN6YIKSIIP433VK6PMMYA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.841671933 (511h58m51.712754968s) [noise=true , address=145.79.227.13:4804 , piece_id=7KOYT64CJDKPR4LGYCN75LDVTTNHNLFAK7JAZI5ETIVLCHKNDPPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842203494 (511h58m51.71328653s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842584951 (511h58m51.713667996s) [noise=true , address=145.79.227.12:4767 , piece_id=6IE2GEK3JYRN7KNZUKA5GSVLU2ACHXT75B6B3NAKOF26NJEBM4CQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842840809 (511h58m51.713923844s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843251670 (511h58m51.714334715s) [noise=true , address=45.91.116.12:4228 , piece_id=U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843530411 (511h58m51.714613446s) [noise=true , address=77.90.31.11:4553 , piece_id=YK6VVJJ2I6Q7XX7WTMF4DGELD5RGORG3EPUMA7E5BYY7Z3JOXLXQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843857985 (511h58m51.71494102s) [noise=true , address=145.79.227.15:4874 , piece_id=QAIG5MOKVWRA7QTYSLDGKSYICFEW3RTCGLPYCPTPLH2W7MFRWI6A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.844225374 (511h58m51.715308409s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5DNPFZFEODNICTDJX7B4SC3EJXR6HF5LA4SJRSCXHF6U6UEFS7A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.844596762 (511h58m51.715679807s) [noise=true , address=145.79.227.36:4517 , piece_id=MYHNRWBYUWGF4XHNGP6MNSR2EX4NKVDK6SERBNER6UEU7Z6RINXA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.846561803 (511h58m51.717644838s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.847861148 (511h58m51.718944183s) [noise=true , address=45.91.116.8:4046 , piece_id=CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.849012220 (511h58m51.720095255s) [noise=true , address=145.79.227.44:4193 , piece_id=I6PZX6YNSSJQ6AIC5TX36FVH7C7TJU6LVDIS5QKFZ2UI3ZF4DAKQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.850316984 (511h58m51.721400029s) [noise=true , address=45.91.116.11:4118 , piece_id=XHE6VOUSZATVMDQKIOXAHL4CLFOFHVXGNN4N3SA6NXKECYZHF5ZA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.851405207 (511h58m51.722488242s) [noise=true , address=77.90.31.15:4697 , piece_id=5AHWFOFJYT3ZT55KTI6VDM4YUPUAFEI5UZRC7XSKZ726AXZ52X3Q, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.852456639 (511h58m51.723539684s) [noise=true , address=145.79.227.47:4267 , piece_id=RQLVEP7QJWEFNAHKMHMMPEAWSDH53CZFDSKGBLEFFTDKLOSNGRPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:53.556959120 (511h58m53.428042165s) [upload=449251]: Next returned nil segment Dec 26 23:27:53.556989167 (511h58m53.428072202s) [upload=449251]: Waiting for error group managing segments... Dec 26 23:27:53.844783598 (511h58m53.715866643s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5DNPFZFEODNICTDJX7B4SC3EJXR6HF5LA4SJRSCXHF6U6UEFS7A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.851886403 (511h58m53.722969448s) [noise=true , address=145.79.227.36:4517 , piece_id=MYHNRWBYUWGF4XHNGP6MNSR2EX4NKVDK6SERBNER6UEU7Z6RINXA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.860937488 (511h58m53.732020533s) [noise=true , address=145.79.227.15:4874 , piece_id=QAIG5MOKVWRA7QTYSLDGKSYICFEW3RTCGLPYCPTPLH2W7MFRWI6A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.867202107 (511h58m53.738285162s) [noise=true , address=145.79.227.12:4767 , piece_id=6IE2GEK3JYRN7KNZUKA5GSVLU2ACHXT75B6B3NAKOF26NJEBM4CQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.867515253 (511h58m53.738598288s) [noise=true , address=145.79.227.47:4267 , piece_id=RQLVEP7QJWEFNAHKMHMMPEAWSDH53CZFDSKGBLEFFTDKLOSNGRPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.869419589 (511h58m53.740502624s) [noise=true , address=145.79.227.13:4804 , piece_id=7KOYT64CJDKPR4LGYCN75LDVTTNHNLFAK7JAZI5ETIVLCHKNDPPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.074603758 (511h58m53.945686794s) [noise=true , address=45.91.116.12:4228 , piece_id=U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.168599989 (511h58m54.039683024s) [noise=true , address=45.91.116.8:4046 , piece_id=CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.188116953 (511h58m54.059199988s) [noise=true , address=77.90.31.19:4842 , piece_id=5J7XL7STKUPXR4UTUGYN24RBQUAQW4QWN6YIKSIIP433VK6PMMYA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.475777599 (511h58m54.346860634s) [noise=true , address=77.90.31.11:4553 , piece_id=YK6VVJJ2I6Q7XX7WTMF4DGELD5RGORG3EPUMA7E5BYY7Z3JOXLXQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.996276903 (511h58m54.867359938s) [noise=true , address=45.91.116.11:4118 , piece_id=XHE6VOUSZATVMDQKIOXAHL4CLFOFHVXGNN4N3SA6NXKECYZHF5ZA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.243663372 (511h58m55.114746407s) [noise=true , address=145.79.227.44:4193 , piece_id=I6PZX6YNSSJQ6AIC5TX36FVH7C7TJU6LVDIS5QKFZ2UI3ZF4DAKQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.486969831 (511h58m55.358052876s) [noise=true , address=77.90.31.15:4697 , piece_id=5AHWFOFJYT3ZT55KTI6VDM4YUPUAFEI5UZRC7XSKZ726AXZ52X3Q, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.486998786 (511h58m55.358081831s) [seg_pos={0 0}, upload=449251]: Segment reached optimal threshold of 13 pieces. Dec 26 23:27:55.487329226 (511h58m55.358412271s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487340747 (511h58m55.358423802s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487395672 (511h58m55.358478717s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487407885 (511h58m55.35849093s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487564944 (511h58m55.358647989s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487572880 (511h58m55.358655925s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487593368 (511h58m55.358676403s) [seg_pos={0 0}, upload=449251]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:27:55.487685253 (511h58m55.358768298s) [upload=449251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:27:55.487691866 (511h58m55.358774911s) [upload=449251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:27:55.779554575 (511h58m55.65063762s) [upload=449251]: Done uploading segments... Dec 26 23:27:55.779577028 (511h58m55.650660063s) [upload=449251]: Upload finished. err: <nil> Duration: 16.343970837s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.896815579s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:28:33 UTC) orbiter warning: context deadline exceeded Duration: 5.896841789s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X.......X..............................X.X.X.X.XX....X.X] duration: 30.003032179s trace: https://jaeger.dev.storj.io/trace/fa7a2ac26cbf8ad FAIL (2025-12-26 23:28:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A) context deadline exceeded ======LOGS====== Dec 26 23:27:51.598399489 (511h58m51.469482524s) [upload=449251]: Starting upload Dec 26 23:27:51.598502145 (511h58m51.46958521s) [upload=449251]: Done starting upload Dec 26 23:27:51.598595983 (511h58m51.469679038s) [upload=449251]: Uploading segments... Dec 26 23:27:51.598696234 (511h58m51.469779279s) [upload=449251]: Got next segment. Inline: false Dec 26 23:27:51.598737111 (511h58m51.469820157s) [upload=449251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:27:51.598743323 (511h58m51.469826378s) [upload=449251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:27:51.841567824 (511h58m51.712650869s) [seg_pos={0 0}, upload=449251]: Begin upload segment... Dec 26 23:27:51.841616236 (511h58m51.712699271s) [seg_pos={0 0}, upload=449251]: Done begin upload segment. Dec 26 23:27:51.841641194 (511h58m51.712724229s) [noise=true , address=77.90.31.19:4842 , piece_id=5J7XL7STKUPXR4UTUGYN24RBQUAQW4QWN6YIKSIIP433VK6PMMYA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.841671933 (511h58m51.712754968s) [noise=true , address=145.79.227.13:4804 , piece_id=7KOYT64CJDKPR4LGYCN75LDVTTNHNLFAK7JAZI5ETIVLCHKNDPPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842203494 (511h58m51.71328653s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842584951 (511h58m51.713667996s) [noise=true , address=145.79.227.12:4767 , piece_id=6IE2GEK3JYRN7KNZUKA5GSVLU2ACHXT75B6B3NAKOF26NJEBM4CQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842840809 (511h58m51.713923844s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843251670 (511h58m51.714334715s) [noise=true , address=45.91.116.12:4228 , piece_id=U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843530411 (511h58m51.714613446s) [noise=true , address=77.90.31.11:4553 , piece_id=YK6VVJJ2I6Q7XX7WTMF4DGELD5RGORG3EPUMA7E5BYY7Z3JOXLXQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843857985 (511h58m51.71494102s) [noise=true , address=145.79.227.15:4874 , piece_id=QAIG5MOKVWRA7QTYSLDGKSYICFEW3RTCGLPYCPTPLH2W7MFRWI6A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.844225374 (511h58m51.715308409s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5DNPFZFEODNICTDJX7B4SC3EJXR6HF5LA4SJRSCXHF6U6UEFS7A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.844596762 (511h58m51.715679807s) [noise=true , address=145.79.227.36:4517 , piece_id=MYHNRWBYUWGF4XHNGP6MNSR2EX4NKVDK6SERBNER6UEU7Z6RINXA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.846561803 (511h58m51.717644838s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.847861148 (511h58m51.718944183s) [noise=true , address=45.91.116.8:4046 , piece_id=CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.849012220 (511h58m51.720095255s) [noise=true , address=145.79.227.44:4193 , piece_id=I6PZX6YNSSJQ6AIC5TX36FVH7C7TJU6LVDIS5QKFZ2UI3ZF4DAKQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.850316984 (511h58m51.721400029s) [noise=true , address=45.91.116.11:4118 , piece_id=XHE6VOUSZATVMDQKIOXAHL4CLFOFHVXGNN4N3SA6NXKECYZHF5ZA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.851405207 (511h58m51.722488242s) [noise=true , address=77.90.31.15:4697 , piece_id=5AHWFOFJYT3ZT55KTI6VDM4YUPUAFEI5UZRC7XSKZ726AXZ52X3Q, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.852456639 (511h58m51.723539684s) [noise=true , address=145.79.227.47:4267 , piece_id=RQLVEP7QJWEFNAHKMHMMPEAWSDH53CZFDSKGBLEFFTDKLOSNGRPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:53.556959120 (511h58m53.428042165s) [upload=449251]: Next returned nil segment Dec 26 23:27:53.556989167 (511h58m53.428072202s) [upload=449251]: Waiting for error group managing segments... Dec 26 23:27:53.844783598 (511h58m53.715866643s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5DNPFZFEODNICTDJX7B4SC3EJXR6HF5LA4SJRSCXHF6U6UEFS7A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.851886403 (511h58m53.722969448s) [noise=true , address=145.79.227.36:4517 , piece_id=MYHNRWBYUWGF4XHNGP6MNSR2EX4NKVDK6SERBNER6UEU7Z6RINXA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.860937488 (511h58m53.732020533s) [noise=true , address=145.79.227.15:4874 , piece_id=QAIG5MOKVWRA7QTYSLDGKSYICFEW3RTCGLPYCPTPLH2W7MFRWI6A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.867202107 (511h58m53.738285162s) [noise=true , address=145.79.227.12:4767 , piece_id=6IE2GEK3JYRN7KNZUKA5GSVLU2ACHXT75B6B3NAKOF26NJEBM4CQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.867515253 (511h58m53.738598288s) [noise=true , address=145.79.227.47:4267 , piece_id=RQLVEP7QJWEFNAHKMHMMPEAWSDH53CZFDSKGBLEFFTDKLOSNGRPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.869419589 (511h58m53.740502624s) [noise=true , address=145.79.227.13:4804 , piece_id=7KOYT64CJDKPR4LGYCN75LDVTTNHNLFAK7JAZI5ETIVLCHKNDPPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.074603758 (511h58m53.945686794s) [noise=true , address=45.91.116.12:4228 , piece_id=U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.168599989 (511h58m54.039683024s) [noise=true , address=45.91.116.8:4046 , piece_id=CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.188116953 (511h58m54.059199988s) [noise=true , address=77.90.31.19:4842 , piece_id=5J7XL7STKUPXR4UTUGYN24RBQUAQW4QWN6YIKSIIP433VK6PMMYA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.475777599 (511h58m54.346860634s) [noise=true , address=77.90.31.11:4553 , piece_id=YK6VVJJ2I6Q7XX7WTMF4DGELD5RGORG3EPUMA7E5BYY7Z3JOXLXQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.996276903 (511h58m54.867359938s) [noise=true , address=45.91.116.11:4118 , piece_id=XHE6VOUSZATVMDQKIOXAHL4CLFOFHVXGNN4N3SA6NXKECYZHF5ZA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.243663372 (511h58m55.114746407s) [noise=true , address=145.79.227.44:4193 , piece_id=I6PZX6YNSSJQ6AIC5TX36FVH7C7TJU6LVDIS5QKFZ2UI3ZF4DAKQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.486969831 (511h58m55.358052876s) [noise=true , address=77.90.31.15:4697 , piece_id=5AHWFOFJYT3ZT55KTI6VDM4YUPUAFEI5UZRC7XSKZ726AXZ52X3Q, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.486998786 (511h58m55.358081831s) [seg_pos={0 0}, upload=449251]: Segment reached optimal threshold of 13 pieces. Dec 26 23:27:55.487329226 (511h58m55.358412271s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487340747 (511h58m55.358423802s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487395672 (511h58m55.358478717s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487407885 (511h58m55.35849093s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487564944 (511h58m55.358647989s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487572880 (511h58m55.358655925s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487593368 (511h58m55.358676403s) [seg_pos={0 0}, upload=449251]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:27:55.487685253 (511h58m55.358768298s) [upload=449251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:27:55.487691866 (511h58m55.358774911s) [upload=449251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:27:55.779554575 (511h58m55.65063762s) [upload=449251]: Done uploading segments... Dec 26 23:27:55.779577028 (511h58m55.650660063s) [upload=449251]: Upload finished. err: <nil> Duration: 30.003056375s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X.......X..............................X.X.X.X.XX....X.] duration: 6.01916603s trace: https://jaeger.dev.storj.io/trace/78807657ce9ceff1 FAIL (2025-12-26 23:27:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.505609383 (511h57m53.376692429s) [upload=449236]: Starting upload Dec 26 23:26:53.505662284 (511h57m53.376745319s) [upload=449236]: Done starting upload Dec 26 23:26:53.505715856 (511h57m53.376798891s) [upload=449236]: Uploading segments... Dec 26 23:26:53.505853358 (511h57m53.376936393s) [upload=449236]: Got next segment. Inline: false Dec 26 23:26:53.505895448 (511h57m53.376978483s) [upload=449236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:26:53.505899556 (511h57m53.376982591s) [upload=449236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:26:53.752751397 (511h57m53.623834432s) [seg_pos={0 0}, upload=449236]: Begin upload segment... Dec 26 23:26:53.752814487 (511h57m53.623897522s) [seg_pos={0 0}, upload=449236]: Done begin upload segment. Dec 26 23:26:53.752856136 (511h57m53.623939171s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.752880012 (511h57m53.623963047s) [noise=true , address=145.79.227.49:4337 , piece_id=YFHMJF7VEX3JQGLT4BTT2EI7CRVBIXVENUIUCB5KJNC6ANKINXCQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.753334968 (511h57m53.624418013s) [noise=true , address=145.79.227.45:4231 , piece_id=BF65NXZ4XB2FMRU3FKKCMWACOU3FIMBAB2ECMSEWAQNDDLHA7KEQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.753677371 (511h57m53.624760406s) [noise=true , address=145.79.227.37:4566 , piece_id=ZGQ64Q3P6C4EYHEO2TV2LNIRFTOWWXMDHNCJUAE3GNVNWP2GZJMQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754145051 (511h57m53.625228086s) [noise=true , address=145.79.227.7:4295 , piece_id=77AWXKRCCCX7LB77SS66PS3YWEUGIKNZQ4MOR24F24EOMQUZKVKQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754551044 (511h57m53.625634089s) [noise=true , address=145.79.227.46:4601 , piece_id=GIZBX6VDLGQEOCS2MXCCZC2G4UQITOVLOL4NTXBWDGI7VPI7M7CA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754867196 (511h57m53.625950231s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.755164061 (511h57m53.626247096s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.756746094 (511h57m53.627829139s) [noise=true , address=145.79.227.48:4301 , piece_id=PHZ3DBRWYZJHT5FQBQJQRGLUUM2K4TQMQUBQ2LV2TQL22XPCPLOA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.757479240 (511h57m53.628562275s) [noise=true , address=145.79.227.11:4729 , piece_id=GJFZCLMQTGIWZGYXJQQ5PDTIHTZBNIF4W6ZXRATAAR5QGT2U4ROA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.759848601 (511h57m53.630931636s) [noise=true , address=45.91.116.5:4298 , piece_id=7NGLZFGZBYZCO55HAD6EEV7D372SWZNLO2VRFQEA75ZIV6ELJEQQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.762664954 (511h57m53.633747989s) [noise=true , address=145.79.227.23:4545 , piece_id=3AB3XAYT55RI4SPNMBBPPOOOG42MJVMFAEVXHUEI5NHHIC63LCFA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.764549372 (511h57m53.635632407s) [noise=true , address=45.91.116.8:4046 , piece_id=Q2TJ5FGZUE4ECRL4SELEMRB62UNMQRYELYI7FTCWS66SO32WEOAA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.766050160 (511h57m53.637133195s) [noise=true , address=77.90.31.14:4662 , piece_id=5IERUYB7PVCISTZAVLQACT256HDXTUWOE7KOEY2JXJ2WTJ43ZMEQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.769742941 (511h57m53.640825976s) [noise=true , address=145.79.227.51:4409 , piece_id=6DA2ZNTV2UEK54EDUAEOJFFABD7JC4GZDH3HQQHT73CU6CPAJ35Q, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.771062714 (511h57m53.642145749s) [noise=true , address=45.91.116.9:4263 , piece_id=OKDFP3FTRPF3LEECF7RPVU4EJT7Z5UYPCVDVPDZ76W7XGAF6YIGQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:55.430505414 (511h57m55.301588449s) [upload=449236]: Next returned nil segment Dec 26 23:26:55.430525082 (511h57m55.301608117s) [upload=449236]: Waiting for error group managing segments... Dec 26 23:26:55.751239376 (511h57m55.622322421s) [noise=true , address=145.79.227.37:4566 , piece_id=ZGQ64Q3P6C4EYHEO2TV2LNIRFTOWWXMDHNCJUAE3GNVNWP2GZJMQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.756350789 (511h57m55.627433824s) [noise=true , address=145.79.227.45:4231 , piece_id=BF65NXZ4XB2FMRU3FKKCMWACOU3FIMBAB2ECMSEWAQNDDLHA7KEQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.759738971 (511h57m55.630822006s) [noise=true , address=145.79.227.49:4337 , piece_id=YFHMJF7VEX3JQGLT4BTT2EI7CRVBIXVENUIUCB5KJNC6ANKINXCQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.759770721 (511h57m55.630853756s) [noise=true , address=145.79.227.46:4601 , piece_id=GIZBX6VDLGQEOCS2MXCCZC2G4UQITOVLOL4NTXBWDGI7VPI7M7CA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.761504331 (511h57m55.632587377s) [noise=true , address=145.79.227.48:4301 , piece_id=PHZ3DBRWYZJHT5FQBQJQRGLUUM2K4TQMQUBQ2LV2TQL22XPCPLOA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.762883748 (511h57m55.633966794s) [noise=true , address=145.79.227.23:4545 , piece_id=3AB3XAYT55RI4SPNMBBPPOOOG42MJVMFAEVXHUEI5NHHIC63LCFA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.773095964 (511h57m55.644178999s) [noise=true , address=145.79.227.7:4295 , piece_id=77AWXKRCCCX7LB77SS66PS3YWEUGIKNZQ4MOR24F24EOMQUZKVKQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.777619698 (511h57m55.648702733s) [noise=true , address=145.79.227.11:4729 , piece_id=GJFZCLMQTGIWZGYXJQQ5PDTIHTZBNIF4W6ZXRATAAR5QGT2U4ROA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.793659660 (511h57m55.664742695s) [noise=true , address=145.79.227.51:4409 , piece_id=6DA2ZNTV2UEK54EDUAEOJFFABD7JC4GZDH3HQQHT73CU6CPAJ35Q, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.977319374 (511h57m55.848402409s) [noise=true , address=77.90.31.14:4662 , piece_id=5IERUYB7PVCISTZAVLQACT256HDXTUWOE7KOEY2JXJ2WTJ43ZMEQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.013973318 (511h57m55.885056353s) [noise=true , address=45.91.116.9:4263 , piece_id=OKDFP3FTRPF3LEECF7RPVU4EJT7Z5UYPCVDVPDZ76W7XGAF6YIGQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.075286767 (511h57m55.946369813s) [noise=true , address=45.91.116.5:4298 , piece_id=7NGLZFGZBYZCO55HAD6EEV7D372SWZNLO2VRFQEA75ZIV6ELJEQQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.081874231 (511h57m55.952957276s) [noise=true , address=45.91.116.8:4046 , piece_id=Q2TJ5FGZUE4ECRL4SELEMRB62UNMQRYELYI7FTCWS66SO32WEOAA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.081891103 (511h57m55.952974148s) [seg_pos={0 0}, upload=449236]: Segment reached optimal threshold of 13 pieces. Dec 26 23:26:56.082082457 (511h57m55.953165502s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 23:26:56.082089861 (511h57m55.953172906s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082388340 (511h57m55.953471375s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection Dec 26 23:26:56.082395604 (511h57m55.953478649s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082479042 (511h57m55.953562097s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 23:26:56.082502688 (511h57m55.953585753s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082558364 (511h57m55.953641409s) [seg_pos={0 0}, upload=449236]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:26:56.082677550 (511h57m55.953760586s) [upload=449236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:26:56.082685796 (511h57m55.953768841s) [upload=449236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:26:56.368370360 (511h57m56.239453395s) [upload=449236]: Done uploading segments... Dec 26 23:26:56.368391740 (511h57m56.239474785s) [upload=449236]: Upload finished. err: <nil> Duration: 6.019180027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X.......X..............................X.X.X.X.XX....X] duration: 30.002523404s trace: https://jaeger.dev.storj.io/trace/78807657ce9ceff1 FAIL (2025-12-26 23:27:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.505609383 (511h57m53.376692429s) [upload=449236]: Starting upload Dec 26 23:26:53.505662284 (511h57m53.376745319s) [upload=449236]: Done starting upload Dec 26 23:26:53.505715856 (511h57m53.376798891s) [upload=449236]: Uploading segments... Dec 26 23:26:53.505853358 (511h57m53.376936393s) [upload=449236]: Got next segment. Inline: false Dec 26 23:26:53.505895448 (511h57m53.376978483s) [upload=449236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:26:53.505899556 (511h57m53.376982591s) [upload=449236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:26:53.752751397 (511h57m53.623834432s) [seg_pos={0 0}, upload=449236]: Begin upload segment... Dec 26 23:26:53.752814487 (511h57m53.623897522s) [seg_pos={0 0}, upload=449236]: Done begin upload segment. Dec 26 23:26:53.752856136 (511h57m53.623939171s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.752880012 (511h57m53.623963047s) [noise=true , address=145.79.227.49:4337 , piece_id=YFHMJF7VEX3JQGLT4BTT2EI7CRVBIXVENUIUCB5KJNC6ANKINXCQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.753334968 (511h57m53.624418013s) [noise=true , address=145.79.227.45:4231 , piece_id=BF65NXZ4XB2FMRU3FKKCMWACOU3FIMBAB2ECMSEWAQNDDLHA7KEQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.753677371 (511h57m53.624760406s) [noise=true , address=145.79.227.37:4566 , piece_id=ZGQ64Q3P6C4EYHEO2TV2LNIRFTOWWXMDHNCJUAE3GNVNWP2GZJMQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754145051 (511h57m53.625228086s) [noise=true , address=145.79.227.7:4295 , piece_id=77AWXKRCCCX7LB77SS66PS3YWEUGIKNZQ4MOR24F24EOMQUZKVKQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754551044 (511h57m53.625634089s) [noise=true , address=145.79.227.46:4601 , piece_id=GIZBX6VDLGQEOCS2MXCCZC2G4UQITOVLOL4NTXBWDGI7VPI7M7CA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754867196 (511h57m53.625950231s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.755164061 (511h57m53.626247096s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.756746094 (511h57m53.627829139s) [noise=true , address=145.79.227.48:4301 , piece_id=PHZ3DBRWYZJHT5FQBQJQRGLUUM2K4TQMQUBQ2LV2TQL22XPCPLOA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.757479240 (511h57m53.628562275s) [noise=true , address=145.79.227.11:4729 , piece_id=GJFZCLMQTGIWZGYXJQQ5PDTIHTZBNIF4W6ZXRATAAR5QGT2U4ROA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.759848601 (511h57m53.630931636s) [noise=true , address=45.91.116.5:4298 , piece_id=7NGLZFGZBYZCO55HAD6EEV7D372SWZNLO2VRFQEA75ZIV6ELJEQQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.762664954 (511h57m53.633747989s) [noise=true , address=145.79.227.23:4545 , piece_id=3AB3XAYT55RI4SPNMBBPPOOOG42MJVMFAEVXHUEI5NHHIC63LCFA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.764549372 (511h57m53.635632407s) [noise=true , address=45.91.116.8:4046 , piece_id=Q2TJ5FGZUE4ECRL4SELEMRB62UNMQRYELYI7FTCWS66SO32WEOAA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.766050160 (511h57m53.637133195s) [noise=true , address=77.90.31.14:4662 , piece_id=5IERUYB7PVCISTZAVLQACT256HDXTUWOE7KOEY2JXJ2WTJ43ZMEQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.769742941 (511h57m53.640825976s) [noise=true , address=145.79.227.51:4409 , piece_id=6DA2ZNTV2UEK54EDUAEOJFFABD7JC4GZDH3HQQHT73CU6CPAJ35Q, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.771062714 (511h57m53.642145749s) [noise=true , address=45.91.116.9:4263 , piece_id=OKDFP3FTRPF3LEECF7RPVU4EJT7Z5UYPCVDVPDZ76W7XGAF6YIGQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:55.430505414 (511h57m55.301588449s) [upload=449236]: Next returned nil segment Dec 26 23:26:55.430525082 (511h57m55.301608117s) [upload=449236]: Waiting for error group managing segments... Dec 26 23:26:55.751239376 (511h57m55.622322421s) [noise=true , address=145.79.227.37:4566 , piece_id=ZGQ64Q3P6C4EYHEO2TV2LNIRFTOWWXMDHNCJUAE3GNVNWP2GZJMQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.756350789 (511h57m55.627433824s) [noise=true , address=145.79.227.45:4231 , piece_id=BF65NXZ4XB2FMRU3FKKCMWACOU3FIMBAB2ECMSEWAQNDDLHA7KEQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.759738971 (511h57m55.630822006s) [noise=true , address=145.79.227.49:4337 , piece_id=YFHMJF7VEX3JQGLT4BTT2EI7CRVBIXVENUIUCB5KJNC6ANKINXCQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.759770721 (511h57m55.630853756s) [noise=true , address=145.79.227.46:4601 , piece_id=GIZBX6VDLGQEOCS2MXCCZC2G4UQITOVLOL4NTXBWDGI7VPI7M7CA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.761504331 (511h57m55.632587377s) [noise=true , address=145.79.227.48:4301 , piece_id=PHZ3DBRWYZJHT5FQBQJQRGLUUM2K4TQMQUBQ2LV2TQL22XPCPLOA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.762883748 (511h57m55.633966794s) [noise=true , address=145.79.227.23:4545 , piece_id=3AB3XAYT55RI4SPNMBBPPOOOG42MJVMFAEVXHUEI5NHHIC63LCFA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.773095964 (511h57m55.644178999s) [noise=true , address=145.79.227.7:4295 , piece_id=77AWXKRCCCX7LB77SS66PS3YWEUGIKNZQ4MOR24F24EOMQUZKVKQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.777619698 (511h57m55.648702733s) [noise=true , address=145.79.227.11:4729 , piece_id=GJFZCLMQTGIWZGYXJQQ5PDTIHTZBNIF4W6ZXRATAAR5QGT2U4ROA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.793659660 (511h57m55.664742695s) [noise=true , address=145.79.227.51:4409 , piece_id=6DA2ZNTV2UEK54EDUAEOJFFABD7JC4GZDH3HQQHT73CU6CPAJ35Q, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.977319374 (511h57m55.848402409s) [noise=true , address=77.90.31.14:4662 , piece_id=5IERUYB7PVCISTZAVLQACT256HDXTUWOE7KOEY2JXJ2WTJ43ZMEQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.013973318 (511h57m55.885056353s) [noise=true , address=45.91.116.9:4263 , piece_id=OKDFP3FTRPF3LEECF7RPVU4EJT7Z5UYPCVDVPDZ76W7XGAF6YIGQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.075286767 (511h57m55.946369813s) [noise=true , address=45.91.116.5:4298 , piece_id=7NGLZFGZBYZCO55HAD6EEV7D372SWZNLO2VRFQEA75ZIV6ELJEQQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.081874231 (511h57m55.952957276s) [noise=true , address=45.91.116.8:4046 , piece_id=Q2TJ5FGZUE4ECRL4SELEMRB62UNMQRYELYI7FTCWS66SO32WEOAA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.081891103 (511h57m55.952974148s) [seg_pos={0 0}, upload=449236]: Segment reached optimal threshold of 13 pieces. Dec 26 23:26:56.082082457 (511h57m55.953165502s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 23:26:56.082089861 (511h57m55.953172906s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082388340 (511h57m55.953471375s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection Dec 26 23:26:56.082395604 (511h57m55.953478649s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082479042 (511h57m55.953562097s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 23:26:56.082502688 (511h57m55.953585753s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082558364 (511h57m55.953641409s) [seg_pos={0 0}, upload=449236]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:26:56.082677550 (511h57m55.953760586s) [upload=449236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:26:56.082685796 (511h57m55.953768841s) [upload=449236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:26:56.368370360 (511h57m56.239453395s) [upload=449236]: Done uploading segments... Dec 26 23:26:56.368391740 (511h57m56.239474785s) [upload=449236]: Upload finished. err: <nil> Duration: 30.00254783s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.886469178s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:27:18 UTC) orbiter warning: context deadline exceeded Duration: 5.886517781s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.......X..............................X.X.X.X.XX....] duration: 8.220331097s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 8.220363228s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.863592649s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:26:08 UTC) orbiter warning: context deadline exceeded Duration: 5.863639629s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.115318089s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:25:16 UTC) orbiter warning: context deadline exceeded Duration: 6.115344057s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.......X.......X..............................X.X.X.X.XX...] duration: 5.955166751s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:25:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 5.955212689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.......X.......X..............................X.X.X.X.XX..] duration: 6.210962483s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 6.210998792s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.921057232s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:24:44 UTC) orbiter warning: context deadline exceeded Duration: 5.921095395s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.845505437s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:24:13 UTC) orbiter warning: context deadline exceeded Duration: 5.84557478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.......X.......X..............................X.X.X.X.XX.] duration: 24.723998936s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:24:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 24.724018442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.......X.......X..............................X.X.X.X.XX] duration: 30.002688518s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:23:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 30.00271085s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.975049525s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:23:25 UTC) orbiter warning: context deadline exceeded Duration: 5.975075563s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.......X.......X..............................X.X.X.X.X] duration: 30.00218s trace: https://jaeger.dev.storj.io/trace/6c9e75584b5f6bf2 FAIL (2025-12-26 23:22:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:22:18.091423839 (511h53m17.962506884s) [upload=449174]: Starting upload Dec 26 23:22:18.091520293 (511h53m17.962603338s) [upload=449174]: Done starting upload Dec 26 23:22:18.091689686 (511h53m17.962772751s) [upload=449174]: Uploading segments... Dec 26 23:22:18.091812088 (511h53m17.962895143s) [upload=449174]: Got next segment. Inline: false Dec 26 23:22:18.091863456 (511h53m17.962946492s) [upload=449174]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:22:18.091869858 (511h53m17.962952894s) [upload=449174]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:22:18.335518279 (511h53m18.206601324s) [seg_pos={0 0}, upload=449174]: Begin upload segment... Dec 26 23:22:18.335564988 (511h53m18.206648023s) [seg_pos={0 0}, upload=449174]: Done begin upload segment. Dec 26 23:22:18.335587381 (511h53m18.206670416s) [noise=true , address=145.79.227.47:4266 , piece_id=LFBF4PKKFEWHYQRGXUEHMWTVHS26OKSAZXWF5YAHJLIRFEUBRKOA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.335673204 (511h53m18.20675625s) [noise=true , address=145.79.227.33:4869 , piece_id=XMFA6KKJV35XYM2JTRBKQBQLHVTTHFCNRC2ZAESBT5FRFES64NRA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.336109385 (511h53m18.20719243s) [noise=true , address=145.79.227.37:4566 , piece_id=CNSCGTMCFEHWELZONKUNC2TEJFNIAH6TC4XWUEZB7A5YW2TTVWOQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.336505929 (511h53m18.207588964s) [noise=true , address=145.79.227.29:4943 , piece_id=25IQXVLTMAOLS3K2VAZ4SL5ECLZ3BPNJTXOPDNM47QTXHX3M5HVA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.336943723 (511h53m18.208026758s) [noise=true , address=45.91.116.7:4010 , piece_id=ILPUYVGX4T6UEGOIFM5PI47YTEH7RJXXU63LYV55PETBK3MA4J6Q, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.337259024 (511h53m18.208342059s) [noise=true , address=45.91.116.4:4939 , piece_id=2H6SKUCMPPP2EJ5XJQWDM3JLXCXTVQD6Q3I3DJTGOZ2LMYIMP6LA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.337645880 (511h53m18.208728925s) [noise=true , address=45.91.116.6:4974 , piece_id=FJ3MQ7CY7AR3562LYRXLK7KD4LKZYQHNZTBU5BKMYUSDMUDOLKWA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.338028559 (511h53m18.209111604s) [noise=true , address=145.79.227.45:4231 , piece_id=AT2QYUKSLZKM7G6MAEI7KACJOYHKKIEPHYSLHPY54R3DAJ3QZBKA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.338555312 (511h53m18.209638347s) [noise=true , address=45.91.116.11:4119 , piece_id=GSGUPXCACFV3QJQCKCYYPE4AJAUVGXZPUTUIAOR3IBO5QXJBHGXQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.339045595 (511h53m18.21012863s) [noise=true , address=45.91.116.5:4297 , piece_id=O4O5Y5Q6DXH4EDMRQGROV6B6RJ4QDPSFMC5HZPA3KXJLHPBMX2NA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.340153275 (511h53m18.21123632s) [noise=true , address=145.79.227.3:4149 , piece_id=IR36U33D2GHCKU4QE7ZNMSXQFDNGLFXRN25YVM7JJ67EMJ5U47OQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.340378865 (511h53m18.2114619s) [noise=true , address=45.91.116.10:4081 , piece_id=OCWPQZMLEQLOUXPTMKZWNAQXN5AYVZ4CKHFYCLPW7QSH7X7UZI4Q, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.341511361 (511h53m18.212594396s) [noise=true , address=145.79.227.48:4301 , piece_id=DGFM47GJEQI77SFLKOV7RR26LQI3OBS6YQAMDZ4QKJV645TQST7A, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.341545345 (511h53m18.21262838s) [noise=true , address=45.91.116.8:4047 , piece_id=C4EHKQYIRN7B4RA6RDOXXQXTV6HL43K6FDWTZGHNGEMLOIXA5RZQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.343464640 (511h53m18.214547685s) [noise=true , address=145.79.227.39:4014 , piece_id=MZFKWP3BSEXAHB5U47QSFETWNCPMKSHVVYVUVAHK773ZGNRZIFEQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.346131518 (511h53m18.217214554s) [noise=true , address=45.91.116.9:4263 , piece_id=UVWN6C3RWYWQJCDYIERJAUA4F76D7EQTXH5GHCJAUMNA4MVN5QMQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:20.056143369 (511h53m19.927226404s) [upload=449174]: Next returned nil segment Dec 26 23:22:20.056161033 (511h53m19.927244078s) [upload=449174]: Waiting for error group managing segments... Dec 26 23:22:20.341483688 (511h53m20.212566723s) [noise=true , address=145.79.227.48:4301 , piece_id=DGFM47GJEQI77SFLKOV7RR26LQI3OBS6YQAMDZ4QKJV645TQST7A, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.346708866 (511h53m20.217791901s) [noise=true , address=145.79.227.45:4231 , piece_id=AT2QYUKSLZKM7G6MAEI7KACJOYHKKIEPHYSLHPY54R3DAJ3QZBKA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.351752971 (511h53m20.222836016s) [noise=true , address=145.79.227.47:4266 , piece_id=LFBF4PKKFEWHYQRGXUEHMWTVHS26OKSAZXWF5YAHJLIRFEUBRKOA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.352979848 (511h53m20.224062893s) [noise=true , address=145.79.227.3:4149 , piece_id=IR36U33D2GHCKU4QE7ZNMSXQFDNGLFXRN25YVM7JJ67EMJ5U47OQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.354083539 (511h53m20.225166594s) [noise=true , address=145.79.227.39:4014 , piece_id=MZFKWP3BSEXAHB5U47QSFETWNCPMKSHVVYVUVAHK773ZGNRZIFEQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.367548749 (511h53m20.238631784s) [noise=true , address=145.79.227.37:4566 , piece_id=CNSCGTMCFEHWELZONKUNC2TEJFNIAH6TC4XWUEZB7A5YW2TTVWOQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.369340511 (511h53m20.240423546s) [noise=true , address=145.79.227.29:4943 , piece_id=25IQXVLTMAOLS3K2VAZ4SL5ECLZ3BPNJTXOPDNM47QTXHX3M5HVA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.562842385 (511h53m20.43392542s) [noise=true , address=45.91.116.6:4974 , piece_id=FJ3MQ7CY7AR3562LYRXLK7KD4LKZYQHNZTBU5BKMYUSDMUDOLKWA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.587894568 (511h53m20.458977603s) [noise=true , address=45.91.116.9:4263 , piece_id=UVWN6C3RWYWQJCDYIERJAUA4F76D7EQTXH5GHCJAUMNA4MVN5QMQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.623504484 (511h53m20.494587529s) [noise=true , address=45.91.116.11:4119 , piece_id=GSGUPXCACFV3QJQCKCYYPE4AJAUVGXZPUTUIAOR3IBO5QXJBHGXQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.647454999 (511h53m20.518538034s) [noise=true , address=45.91.116.5:4297 , piece_id=O4O5Y5Q6DXH4EDMRQGROV6B6RJ4QDPSFMC5HZPA3KXJLHPBMX2NA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.650344832 (511h53m20.521427877s) [noise=true , address=45.91.116.4:4939 , piece_id=2H6SKUCMPPP2EJ5XJQWDM3JLXCXTVQD6Q3I3DJTGOZ2LMYIMP6LA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.694666056 (511h53m20.565749091s) [noise=true , address=45.91.116.8:4047 , piece_id=C4EHKQYIRN7B4RA6RDOXXQXTV6HL43K6FDWTZGHNGEMLOIXA5RZQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.694686314 (511h53m20.565769349s) [seg_pos={0 0}, upload=449174]: Segment reached optimal threshold of 13 pieces. Dec 26 23:22:20.694876687 (511h53m20.565959722s) [noise=true , address=145.79.227.33:4869 , piece_id=XMFA6KKJV35XYM2JTRBKQBQLHVTTHFCNRC2ZAESBT5FRFES64NRA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 26 23:22:20.694882929 (511h53m20.565965964s) [noise=true , address=145.79.227.33:4869 , piece_id=XMFA6KKJV35XYM2JTRBKQBQLHVTTHFCNRC2ZAESBT5FRFES64NRA, seg_pos={0 0}, upload=449174]: Long tail cancellation due to optimal threshold Dec 26 23:22:20.695113168 (511h53m20.566196213s) [noise=true , address=45.91.116.7:4010 , piece_id=ILPUYVGX4T6UEGOIFM5PI47YTEH7RJXXU63LYV55PETBK3MA4J6Q, seg_pos={0 0}, upload=449174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 23:22:20.695141471 (511h53m20.566224506s) [noise=true , address=45.91.116.7:4010 , piece_id=ILPUYVGX4T6UEGOIFM5PI47YTEH7RJXXU63LYV55PETBK3MA4J6Q, seg_pos={0 0}, upload=449174]: Long tail cancellation due to optimal threshold Dec 26 23:22:20.695170616 (511h53m20.566253691s) [noise=true , address=45.91.116.10:4081 , piece_id=OCWPQZMLEQLOUXPTMKZWNAQXN5AYVZ4CKHFYCLPW7QSH7X7UZI4Q, seg_pos={0 0}, upload=449174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 23:22:20.695181157 (511h53m20.566264202s) [noise=true , address=45.91.116.10:4081 , piece_id=OCWPQZMLEQLOUXPTMKZWNAQXN5AYVZ4CKHFYCLPW7QSH7X7UZI4Q, seg_pos={0 0}, upload=449174]: Long tail cancellation due to optimal threshold Dec 26 23:22:20.695204391 (511h53m20.566287426s) [seg_pos={0 0}, upload=449174]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:22:20.695278321 (511h53m20.566361356s) [upload=449174]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:22:20.695285386 (511h53m20.566368441s) [upload=449174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:22:20.980288329 (511h53m20.851371374s) [upload=449174]: Done uploading segments... Dec 26 23:22:20.980312145 (511h53m20.85139518s) [upload=449174]: Upload finished. err: <nil> Duration: 30.002193757s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.031776486s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:22:44 UTC) orbiter warning: context deadline exceeded Duration: 6.031804759s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.925403724s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:21:37 UTC) orbiter warning: context deadline exceeded Duration: 5.925426617s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)