Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX.] duration: 8.895001563s trace: https://jaeger.dev.storj.io/trace/3c8fe8e9719880ca FAIL (2025-12-29 17:02:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q) context deadline exceeded ======LOGS====== 9 17:00:45.539911988 (577h31m45.410995023s) [upload=506280]: Starting upload Dec 29 17:00:45.539986039 (577h31m45.411069074s) [upload=506280]: Done starting upload Dec 29 17:00:45.540046224 (577h31m45.411129279s) [upload=506280]: Uploading segments... Dec 29 17:00:45.540180901 (577h31m45.411263936s) [upload=506280]: Got next segment. Inline: false Dec 29 17:00:45.540216719 (577h31m45.411299754s) [upload=506280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:00:45.540221097 (577h31m45.411304142s) [upload=506280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:00:45.789919108 (577h31m45.661002143s) [seg_pos={0 0}, upload=506280]: Begin upload segment... Dec 29 17:00:45.789961138 (577h31m45.661044173s) [seg_pos={0 0}, upload=506280]: Done begin upload segment. Dec 29 17:00:45.789987519 (577h31m45.661070564s) [noise=true , address=45.91.116.9:4264 , piece_id=QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790033055 (577h31m45.6611161s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790546112 (577h31m45.661629147s) [noise=true , address=145.79.227.26:4653 , piece_id=UH3RSFNA6I4R4DHBXRC7266TSNPVFP5FBY4JOD56OGK4Z7ZEFNLQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790949070 (577h31m45.662032105s) [noise=true , address=145.79.227.18:4006 , piece_id=KGCVAYT6UEB2Y6CXM33F6NQYZRJB2YQ4B53CQZ7QK7JVK627TTIQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.791365723 (577h31m45.662448768s) [noise=true , address=145.79.227.50:4373 , piece_id=AIWX2TOHKOHPAX6YW7EZ2E6CNV2HM45CPWJHLGWBZMLCHQPFCYLA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.791818365 (577h31m45.66290141s) [noise=true , address=45.91.116.11:4117 , piece_id=ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.792248674 (577h31m45.663331719s) [noise=true , address=145.79.227.3:4151 , piece_id=XFREHRJ7A6YKASFUFASAWKKEDOORPGMHU6TVPCDPDT66CHUUCSRQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.792710193 (577h31m45.663793238s) [noise=true , address=145.79.227.22:4509 , piece_id=AR72NLFK6NOHS6TOYSZML673GORGS3X5Z72MVP2VPAAU34AKV4YA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.794496304 (577h31m45.66557934s) [noise=true , address=45.91.116.4:4940 , piece_id=6437A5YQW6Y2INQM66AFSBGHO45YLNYB7QYO3R6EW4KKD4JMRPIQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.795684126 (577h31m45.666767171s) [noise=true , address=145.79.227.8:4329 , piece_id=AUMS5RFNJA3NAYHS6AUVDGC3TO4ZFGMWHKESNGBVBEDVZ73JF7DQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.797240760 (577h31m45.668323805s) [noise=true , address=145.79.227.15:4873 , piece_id=76NDNBEWTROGTCNPBG56AZUSPNGC33V7UHDCC2OMD322JVKIWNLQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.800093171 (577h31m45.671176206s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.801977329 (577h31m45.673060374s) [noise=true , address=145.79.227.16:4910 , piece_id=OZYALE4IHKIQ4MZGMER6QXPX6DFZAD4IOK3C5M7L2KM5IIQZRCOQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.804772762 (577h31m45.675855808s) [noise=true , address=145.79.227.42:4123 , piece_id=OAOIID5WANPGNHPWOVGSJJSZ5W2PTOFPE65EPINUNT6UMV5SE2UQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.806558573 (577h31m45.677641608s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.811097495 (577h31m45.68218053s) [noise=true , address=145.79.227.41:4085 , piece_id=54A7KBP7QJQ5RYXSN4WLF4H6FH27XDO6HGVGSKXYDDELGUDF4X4Q, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:47.489597096 (577h31m47.360680131s) [upload=506280]: Next returned nil segment Dec 29 17:00:47.489610752 (577h31m47.360693787s) [upload=506280]: Waiting for error group managing segments... Dec 29 17:00:47.769739099 (577h31m47.640822144s) [noise=true , address=145.79.227.8:4329 , piece_id=AUMS5RFNJA3NAYHS6AUVDGC3TO4ZFGMWHKESNGBVBEDVZ73JF7DQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.770748601 (577h31m47.641831646s) [noise=true , address=145.79.227.15:4873 , piece_id=76NDNBEWTROGTCNPBG56AZUSPNGC33V7UHDCC2OMD322JVKIWNLQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.785560665 (577h31m47.65664371s) [noise=true , address=145.79.227.50:4373 , piece_id=AIWX2TOHKOHPAX6YW7EZ2E6CNV2HM45CPWJHLGWBZMLCHQPFCYLA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.785772348 (577h31m47.656855383s) [noise=true , address=145.79.227.3:4151 , piece_id=XFREHRJ7A6YKASFUFASAWKKEDOORPGMHU6TVPCDPDT66CHUUCSRQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.799070519 (577h31m47.670153564s) [noise=true , address=145.79.227.22:4509 , piece_id=AR72NLFK6NOHS6TOYSZML673GORGS3X5Z72MVP2VPAAU34AKV4YA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.801725265 (577h31m47.67280831s) [noise=true , address=145.79.227.16:4910 , piece_id=OZYALE4IHKIQ4MZGMER6QXPX6DFZAD4IOK3C5M7L2KM5IIQZRCOQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.803057211 (577h31m47.674140256s) [noise=true , address=145.79.227.26:4653 , piece_id=UH3RSFNA6I4R4DHBXRC7266TSNPVFP5FBY4JOD56OGK4Z7ZEFNLQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.814788328 (577h31m47.685871373s) [noise=true , address=145.79.227.42:4123 , piece_id=OAOIID5WANPGNHPWOVGSJJSZ5W2PTOFPE65EPINUNT6UMV5SE2UQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.922372444 (577h31m47.793455479s) [noise=true , address=145.79.227.18:4006 , piece_id=KGCVAYT6UEB2Y6CXM33F6NQYZRJB2YQ4B53CQZ7QK7JVK627TTIQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.046216582 (577h31m47.917299627s) [noise=true , address=45.91.116.9:4264 , piece_id=QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.051177748 (577h31m47.922260784s) [noise=true , address=45.91.116.4:4940 , piece_id=6437A5YQW6Y2INQM66AFSBGHO45YLNYB7QYO3R6EW4KKD4JMRPIQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.057471553 (577h31m47.928554598s) [noise=true , address=45.91.116.11:4117 , piece_id=ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.229734237 (577h31m48.100817282s) [noise=true , address=145.79.227.41:4085 , piece_id=54A7KBP7QJQ5RYXSN4WLF4H6FH27XDO6HGVGSKXYDDELGUDF4X4Q, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.229759335 (577h31m48.10084238s) [seg_pos={0 0}, upload=506280]: Segment reached optimal threshold of 13 pieces. Dec 29 17:00:48.230163884 (577h31m48.101246939s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 29 17:00:48.230174205 (577h31m48.10125726s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230537917 (577h31m48.101620952s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 17:00:48.230547996 (577h31m48.101631031s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230712549 (577h31m48.101795594s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection Dec 29 17:00:48.230722859 (577h31m48.101805904s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230747917 (577h31m48.101830952s) [seg_pos={0 0}, upload=506280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:00:48.230825324 (577h31m48.101908369s) [upload=506280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:00:48.230831486 (577h31m48.101914531s) [upload=506280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:00:48.532400788 (577h31m48.403483843s) [upload=506280]: Done uploading segments... Dec 29 17:00:48.532425275 (577h31m48.40350831s) [upload=506280]: Upload finished. err: <nil> Duration: 8.895028956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X......................X.....] duration: 37.445721419s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 17:01:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 37.445740014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.] duration: 5.00155668s trace: https://jaeger.dev.storj.io/trace/176351fc92e5fa36 FAIL (2025-12-29 17:01:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001584113s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X] duration: 5.001987411s trace: https://jaeger.dev.storj.io/trace/176351fc92e5fa36 FAIL (2025-12-29 17:01:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002013911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......X......................X....] duration: 42.817902127s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 17:01:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 42.817943855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX] duration: 30.003020793s trace: https://jaeger.dev.storj.io/trace/3c8fe8e9719880ca FAIL (2025-12-29 17:01:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q) context deadline exceeded ======LOGS====== 9 17:00:45.539911988 (577h31m45.410995023s) [upload=506280]: Starting upload Dec 29 17:00:45.539986039 (577h31m45.411069074s) [upload=506280]: Done starting upload Dec 29 17:00:45.540046224 (577h31m45.411129279s) [upload=506280]: Uploading segments... Dec 29 17:00:45.540180901 (577h31m45.411263936s) [upload=506280]: Got next segment. Inline: false Dec 29 17:00:45.540216719 (577h31m45.411299754s) [upload=506280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:00:45.540221097 (577h31m45.411304142s) [upload=506280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:00:45.789919108 (577h31m45.661002143s) [seg_pos={0 0}, upload=506280]: Begin upload segment... Dec 29 17:00:45.789961138 (577h31m45.661044173s) [seg_pos={0 0}, upload=506280]: Done begin upload segment. Dec 29 17:00:45.789987519 (577h31m45.661070564s) [noise=true , address=45.91.116.9:4264 , piece_id=QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790033055 (577h31m45.6611161s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790546112 (577h31m45.661629147s) [noise=true , address=145.79.227.26:4653 , piece_id=UH3RSFNA6I4R4DHBXRC7266TSNPVFP5FBY4JOD56OGK4Z7ZEFNLQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790949070 (577h31m45.662032105s) [noise=true , address=145.79.227.18:4006 , piece_id=KGCVAYT6UEB2Y6CXM33F6NQYZRJB2YQ4B53CQZ7QK7JVK627TTIQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.791365723 (577h31m45.662448768s) [noise=true , address=145.79.227.50:4373 , piece_id=AIWX2TOHKOHPAX6YW7EZ2E6CNV2HM45CPWJHLGWBZMLCHQPFCYLA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.791818365 (577h31m45.66290141s) [noise=true , address=45.91.116.11:4117 , piece_id=ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.792248674 (577h31m45.663331719s) [noise=true , address=145.79.227.3:4151 , piece_id=XFREHRJ7A6YKASFUFASAWKKEDOORPGMHU6TVPCDPDT66CHUUCSRQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.792710193 (577h31m45.663793238s) [noise=true , address=145.79.227.22:4509 , piece_id=AR72NLFK6NOHS6TOYSZML673GORGS3X5Z72MVP2VPAAU34AKV4YA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.794496304 (577h31m45.66557934s) [noise=true , address=45.91.116.4:4940 , piece_id=6437A5YQW6Y2INQM66AFSBGHO45YLNYB7QYO3R6EW4KKD4JMRPIQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.795684126 (577h31m45.666767171s) [noise=true , address=145.79.227.8:4329 , piece_id=AUMS5RFNJA3NAYHS6AUVDGC3TO4ZFGMWHKESNGBVBEDVZ73JF7DQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.797240760 (577h31m45.668323805s) [noise=true , address=145.79.227.15:4873 , piece_id=76NDNBEWTROGTCNPBG56AZUSPNGC33V7UHDCC2OMD322JVKIWNLQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.800093171 (577h31m45.671176206s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.801977329 (577h31m45.673060374s) [noise=true , address=145.79.227.16:4910 , piece_id=OZYALE4IHKIQ4MZGMER6QXPX6DFZAD4IOK3C5M7L2KM5IIQZRCOQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.804772762 (577h31m45.675855808s) [noise=true , address=145.79.227.42:4123 , piece_id=OAOIID5WANPGNHPWOVGSJJSZ5W2PTOFPE65EPINUNT6UMV5SE2UQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.806558573 (577h31m45.677641608s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.811097495 (577h31m45.68218053s) [noise=true , address=145.79.227.41:4085 , piece_id=54A7KBP7QJQ5RYXSN4WLF4H6FH27XDO6HGVGSKXYDDELGUDF4X4Q, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:47.489597096 (577h31m47.360680131s) [upload=506280]: Next returned nil segment Dec 29 17:00:47.489610752 (577h31m47.360693787s) [upload=506280]: Waiting for error group managing segments... Dec 29 17:00:47.769739099 (577h31m47.640822144s) [noise=true , address=145.79.227.8:4329 , piece_id=AUMS5RFNJA3NAYHS6AUVDGC3TO4ZFGMWHKESNGBVBEDVZ73JF7DQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.770748601 (577h31m47.641831646s) [noise=true , address=145.79.227.15:4873 , piece_id=76NDNBEWTROGTCNPBG56AZUSPNGC33V7UHDCC2OMD322JVKIWNLQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.785560665 (577h31m47.65664371s) [noise=true , address=145.79.227.50:4373 , piece_id=AIWX2TOHKOHPAX6YW7EZ2E6CNV2HM45CPWJHLGWBZMLCHQPFCYLA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.785772348 (577h31m47.656855383s) [noise=true , address=145.79.227.3:4151 , piece_id=XFREHRJ7A6YKASFUFASAWKKEDOORPGMHU6TVPCDPDT66CHUUCSRQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.799070519 (577h31m47.670153564s) [noise=true , address=145.79.227.22:4509 , piece_id=AR72NLFK6NOHS6TOYSZML673GORGS3X5Z72MVP2VPAAU34AKV4YA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.801725265 (577h31m47.67280831s) [noise=true , address=145.79.227.16:4910 , piece_id=OZYALE4IHKIQ4MZGMER6QXPX6DFZAD4IOK3C5M7L2KM5IIQZRCOQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.803057211 (577h31m47.674140256s) [noise=true , address=145.79.227.26:4653 , piece_id=UH3RSFNA6I4R4DHBXRC7266TSNPVFP5FBY4JOD56OGK4Z7ZEFNLQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.814788328 (577h31m47.685871373s) [noise=true , address=145.79.227.42:4123 , piece_id=OAOIID5WANPGNHPWOVGSJJSZ5W2PTOFPE65EPINUNT6UMV5SE2UQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.922372444 (577h31m47.793455479s) [noise=true , address=145.79.227.18:4006 , piece_id=KGCVAYT6UEB2Y6CXM33F6NQYZRJB2YQ4B53CQZ7QK7JVK627TTIQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.046216582 (577h31m47.917299627s) [noise=true , address=45.91.116.9:4264 , piece_id=QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.051177748 (577h31m47.922260784s) [noise=true , address=45.91.116.4:4940 , piece_id=6437A5YQW6Y2INQM66AFSBGHO45YLNYB7QYO3R6EW4KKD4JMRPIQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.057471553 (577h31m47.928554598s) [noise=true , address=45.91.116.11:4117 , piece_id=ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.229734237 (577h31m48.100817282s) [noise=true , address=145.79.227.41:4085 , piece_id=54A7KBP7QJQ5RYXSN4WLF4H6FH27XDO6HGVGSKXYDDELGUDF4X4Q, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.229759335 (577h31m48.10084238s) [seg_pos={0 0}, upload=506280]: Segment reached optimal threshold of 13 pieces. Dec 29 17:00:48.230163884 (577h31m48.101246939s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 29 17:00:48.230174205 (577h31m48.10125726s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230537917 (577h31m48.101620952s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 17:00:48.230547996 (577h31m48.101631031s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230712549 (577h31m48.101795594s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection Dec 29 17:00:48.230722859 (577h31m48.101805904s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230747917 (577h31m48.101830952s) [seg_pos={0 0}, upload=506280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:00:48.230825324 (577h31m48.101908369s) [upload=506280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:00:48.230831486 (577h31m48.101914531s) [upload=506280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:00:48.532400788 (577h31m48.403483843s) [upload=506280]: Done uploading segments... Dec 29 17:00:48.532425275 (577h31m48.40350831s) [upload=506280]: Upload finished. err: <nil> Duration: 30.003044027s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........] duration: 4.013465481s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 17:00:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.013493124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XX] duration: 30.004039001s trace: https://jaeger.dev.storj.io/trace/74c2217b44eef6ef FAIL (2025-12-29 17:00:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:59:53.233034632 (577h30m53.104117667s) [upload=506268]: Starting upload Dec 29 16:59:53.233085378 (577h30m53.104168413s) [upload=506268]: Done starting upload Dec 29 16:59:53.233276112 (577h30m53.104359147s) [upload=506268]: Uploading segments... Dec 29 16:59:53.233340674 (577h30m53.104423709s) [upload=506268]: Got next segment. Inline: false Dec 29 16:59:53.233363848 (577h30m53.104446893s) [upload=506268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:59:53.233367756 (577h30m53.104450801s) [upload=506268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:59:53.483105403 (577h30m53.354188458s) [seg_pos={0 0}, upload=506268]: Begin upload segment... Dec 29 16:59:53.483216204 (577h30m53.354299249s) [seg_pos={0 0}, upload=506268]: Done begin upload segment. Dec 29 16:59:53.483279735 (577h30m53.35436277s) [noise=true , address=145.79.227.18:4006 , piece_id=H2X27FXNN2XUTSQC6GCIH6TUHDEEDVWXOFHPIYBV3PHHQZJ2HEWA, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.483299744 (577h30m53.354382779s) [noise=true , address=77.90.31.13:4627 , piece_id=GEJBD7ZHGZVMT22QBZHXCDEL5D77A6LV2SEIOXXSJ23EUETO6PNQ, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.483871532 (577h30m53.354954577s) [noise=true , address=145.79.227.38:4977 , piece_id=ZFOC53FQBOVBSGRA6X5YWBMPFJWORFVQBGGP3HMPHJ4Q2TXU645Q, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.484337179 (577h30m53.355420214s) [noise=true , address=145.79.227.24:4582 , piece_id=CPRSMVV26WBOWEEJ7C7TSCEW5HZJEI3UT2CULVHYN7WMUFAJYEQA, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.484820218 (577h30m53.355903263s) [noise=true , address=145.79.227.50:4374 , piece_id=JCRL5LS3SOWG64NPJ6T7CSPCKECBUL7NAN46IPSE56GF3XIC4BQA, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.485742886 (577h30m53.356825921s) [noise=true , address=45.91.116.11:4118 , piece_id=JFYVFKCNY2M4FB5H24S7QGHUXY4SYNSAPUPIIVL2JOKKT5WFHYEA, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.486845606 (577h30m53.357928651s) [noise=true , address=145.79.227.46:4601 , piece_id=LQG5ZN5OO3BMC5ONRSSE3Q5DB4AFNPOOREIVSWT2T2UHKEN5WA7Q, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.487186444 (577h30m53.358269479s) [noise=true , address=77.90.31.16:4735 , piece_id=7NFOMVO2ZSWEYMJCYTZOH5IATG4VZ7BTDXLEG5H423Y7FK73AWMQ, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.487672079 (577h30m53.358755124s) [noise=true , address=145.79.227.12:4767 , piece_id=SMCUUHQZXVXLULGFJP54VR6HSPWFLCH6YZQVIWFEG5AX6LTGHJXA, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.489032379 (577h30m53.360115434s) [noise=true , address=45.91.116.9:4264 , piece_id=3ZNDLOKYZXYEWCG46R6Y2UXAHD3KFK3B7FLV6PH37HGYTERDLQUQ, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.489428193 (577h30m53.360511228s) [noise=true , address=145.79.227.52:4447 , piece_id=3K224VFFN4T7CWKJHRWG5MFNCJZIQTWHGV6FF2EFOQBLTODQRUCA, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.490848117 (577h30m53.361931152s) [noise=true , address=145.79.227.22:4510 , piece_id=VP6EQRP77IOQ6HT5QIBYNHZQ67TUJ3DN7P6B4IIJZQXP3ABXBYJA, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.491764482 (577h30m53.362847547s) [noise=true , address=145.79.227.41:4087 , piece_id=NLAWMPTQO7A33ORJ6HKTR5MNXSHXLN6FEE522TERGATJE4MAFPIA, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.492279712 (577h30m53.363362747s) [noise=true , address=45.91.116.12:4225 , piece_id=QG6BAVEL55X4C6EEEBGSYSEFX2FPYEHYAWANCCGLZXIZ3KEDP67A, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.493032806 (577h30m53.364115842s) [noise=true , address=45.91.116.5:4298 , piece_id=4MCWL56JNSM7I2N7HEDEIYEQS34KOFCRZR3VLH727YNQ3LFDBVAQ, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:53.493914315 (577h30m53.36499735s) [noise=true , address=145.79.227.51:4410 , piece_id=4THIXE4JHBIKO4UB6T7GWLBXFDRMCLTYWC3OTPZNCMA6CNTZZK4A, seg_pos={0 0}, upload=506268]: Uploading piece... Dec 29 16:59:55.198371152 (577h30m55.069454187s) [upload=506268]: Next returned nil segment Dec 29 16:59:55.198384989 (577h30m55.069468024s) [upload=506268]: Waiting for error group managing segments... Dec 29 16:59:55.478823205 (577h30m55.34990624s) [noise=true , address=145.79.227.51:4410 , piece_id=4THIXE4JHBIKO4UB6T7GWLBXFDRMCLTYWC3OTPZNCMA6CNTZZK4A, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.484690096 (577h30m55.355773141s) [noise=true , address=145.79.227.38:4977 , piece_id=ZFOC53FQBOVBSGRA6X5YWBMPFJWORFVQBGGP3HMPHJ4Q2TXU645Q, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.485336016 (577h30m55.356419061s) [noise=true , address=145.79.227.22:4510 , piece_id=VP6EQRP77IOQ6HT5QIBYNHZQ67TUJ3DN7P6B4IIJZQXP3ABXBYJA, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.486718298 (577h30m55.357801343s) [noise=true , address=145.79.227.46:4601 , piece_id=LQG5ZN5OO3BMC5ONRSSE3Q5DB4AFNPOOREIVSWT2T2UHKEN5WA7Q, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.489182220 (577h30m55.360265255s) [noise=true , address=145.79.227.18:4006 , piece_id=H2X27FXNN2XUTSQC6GCIH6TUHDEEDVWXOFHPIYBV3PHHQZJ2HEWA, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.506409924 (577h30m55.377492959s) [noise=true , address=145.79.227.52:4447 , piece_id=3K224VFFN4T7CWKJHRWG5MFNCJZIQTWHGV6FF2EFOQBLTODQRUCA, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.507252909 (577h30m55.378335944s) [noise=true , address=145.79.227.24:4582 , piece_id=CPRSMVV26WBOWEEJ7C7TSCEW5HZJEI3UT2CULVHYN7WMUFAJYEQA, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.510195362 (577h30m55.381278407s) [noise=true , address=145.79.227.41:4087 , piece_id=NLAWMPTQO7A33ORJ6HKTR5MNXSHXLN6FEE522TERGATJE4MAFPIA, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.514880523 (577h30m55.385963559s) [noise=true , address=145.79.227.50:4374 , piece_id=JCRL5LS3SOWG64NPJ6T7CSPCKECBUL7NAN46IPSE56GF3XIC4BQA, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.521686803 (577h30m55.392769838s) [noise=true , address=145.79.227.12:4767 , piece_id=SMCUUHQZXVXLULGFJP54VR6HSPWFLCH6YZQVIWFEG5AX6LTGHJXA, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.664014777 (577h30m55.535097822s) [noise=true , address=77.90.31.16:4735 , piece_id=7NFOMVO2ZSWEYMJCYTZOH5IATG4VZ7BTDXLEG5H423Y7FK73AWMQ, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.733100797 (577h30m55.604183832s) [noise=true , address=45.91.116.12:4225 , piece_id=QG6BAVEL55X4C6EEEBGSYSEFX2FPYEHYAWANCCGLZXIZ3KEDP67A, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.745816109 (577h30m55.616899154s) [noise=true , address=45.91.116.9:4264 , piece_id=3ZNDLOKYZXYEWCG46R6Y2UXAHD3KFK3B7FLV6PH37HGYTERDLQUQ, seg_pos={0 0}, upload=506268]: Done uploading piece. err: <nil> Dec 29 16:59:55.745834614 (577h30m55.616917649s) [seg_pos={0 0}, upload=506268]: Segment reached optimal threshold of 13 pieces. Dec 29 16:59:55.746068359 (577h30m55.617151394s) [noise=true , address=45.91.116.11:4118 , piece_id=JFYVFKCNY2M4FB5H24S7QGHUXY4SYNSAPUPIIVL2JOKKT5WFHYEA, seg_pos={0 0}, upload=506268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:42126->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 16:59:55.746075082 (577h30m55.617158117s) [noise=true , address=45.91.116.11:4118 , piece_id=JFYVFKCNY2M4FB5H24S7QGHUXY4SYNSAPUPIIVL2JOKKT5WFHYEA, seg_pos={0 0}, upload=506268]: Long tail cancellation due to optimal threshold Dec 29 16:59:55.746085622 (577h30m55.617168657s) [noise=true , address=77.90.31.13:4627 , piece_id=GEJBD7ZHGZVMT22QBZHXCDEL5D77A6LV2SEIOXXSJ23EUETO6PNQ, seg_pos={0 0}, upload=506268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 16:59:55.746094218 (577h30m55.617177253s) [noise=true , address=77.90.31.13:4627 , piece_id=GEJBD7ZHGZVMT22QBZHXCDEL5D77A6LV2SEIOXXSJ23EUETO6PNQ, seg_pos={0 0}, upload=506268]: Long tail cancellation due to optimal threshold Dec 29 16:59:55.746189971 (577h30m55.617273006s) [noise=true , address=45.91.116.5:4298 , piece_id=4MCWL56JNSM7I2N7HEDEIYEQS34KOFCRZR3VLH727YNQ3LFDBVAQ, seg_pos={0 0}, upload=506268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 16:59:55.746195982 (577h30m55.617279027s) [noise=true , address=45.91.116.5:4298 , piece_id=4MCWL56JNSM7I2N7HEDEIYEQS34KOFCRZR3VLH727YNQ3LFDBVAQ, seg_pos={0 0}, upload=506268]: Long tail cancellation due to optimal threshold Dec 29 16:59:55.746246729 (577h30m55.617329764s) [seg_pos={0 0}, upload=506268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:59:55.746302094 (577h30m55.617385129s) [upload=506268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:59:55.746306162 (577h30m55.617389197s) [upload=506268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:59:56.037398505 (577h30m55.90848154s) [upload=506268]: Done uploading segments... Dec 29 16:59:56.037421498 (577h30m55.908504543s) [upload=506268]: Upload finished. err: <nil> Duration: 30.004052096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......X......................X...] duration: 25.568833912s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 17:00:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 25.568863466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....X] duration: 30.00266089s trace: https://jaeger.dev.storj.io/trace/7e634fa040b26385 FAIL (2025-12-29 16:59:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s... Dec 29 16:59:22.973700466 (577h30m22.844783501s) [upload=506260]: Got next segment. Inline: false Dec 29 16:59:22.973732117 (577h30m22.844815152s) [upload=506260]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:59:22.973769257 (577h30m22.844852302s) [upload=506260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:59:23.216756431 (577h30m23.087839466s) [seg_pos={0 0}, upload=506260]: Begin upload segment... Dec 29 16:59:23.216806426 (577h30m23.087889461s) [seg_pos={0 0}, upload=506260]: Done begin upload segment. Dec 29 16:59:23.216836904 (577h30m23.087919949s) [noise=true , address=145.79.227.8:4329 , piece_id=DHT2VOOEJIYCIJMRUUO5MPDJDYMBWWFDMIVL7N4ANQQNEIBX6SBQ, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.216844769 (577h30m23.087927804s) [noise=true , address=45.91.116.13:4155 , piece_id=LSEQKHCVE7RFLCTVOCL3OAVGINDPXMPH6IQBWFCG4UA5CONBUCNA, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.217403052 (577h30m23.088486087s) [noise=true , address=45.91.116.12:4225 , piece_id=POF7HZELBAHMX4CD2WZACINZNOJ3L4ON6HATYYOSMVUBHORUJD3Q, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.217472234 (577h30m23.088555269s) [noise=true , address=145.79.227.32:4836 , piece_id=AR27H3RUGPXEUURKFS4NQM354ZC4EKJVPVCRLRVVN7TVC452MZ4A, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.217819305 (577h30m23.08890234s) [noise=true , address=77.90.31.15:4697 , piece_id=XSYVFBIDNX3OWGMJ2KMFFY6XMW5K7OGB5F6UBQXCNNJERSZBB7WA, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.218153631 (577h30m23.089236676s) [noise=true , address=145.79.227.5:4221 , piece_id=GLUVY7N6KIKZY5MPGDGJK7RLUMCTZABTDZNMXKYUFFYWKNCZUM7Q, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.218731191 (577h30m23.089814236s) [noise=true , address=145.79.227.48:4301 , piece_id=NSGQ4MHZLZ2V74BMHBWNYVOXGZOLWET2XXEDNU76JGKLPOFPQIYA, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.218791336 (577h30m23.089874371s) [noise=true , address=145.79.227.44:4194 , piece_id=RBO2B6FMLPW4K2IQWI4IR4ICDIJEEJO2ROSGUPSCKVA3PJLP2XJA, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.219375638 (577h30m23.090458683s) [noise=true , address=77.90.31.17:4773 , piece_id=SJMPVHE5NUABVLPQKKGN5FYD6LNOPRBSHA7UL6CP6A4ZHG6OAFDQ, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.219557184 (577h30m23.090640219s) [noise=true , address=77.90.31.16:4735 , piece_id=U72ALMBCMK3FV6JFQZ2BY7PGU5ET4ONMRS46LVDSFWKM2LGEZMTQ, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.220128051 (577h30m23.091211086s) [noise=true , address=145.79.227.6:4257 , piece_id=AGTOP6RMQ4VHID6RRZ2JFRAHYVA5A3KGQ72YUUD5J6SOFNSHDAMQ, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.220680022 (577h30m23.091763057s) [noise=true , address=145.79.227.47:4266 , piece_id=S76BYQ256IKABUXGWMCEI3PKEWEL3YWAS7NFSQGOPQWYGI3MPT7A, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.222285719 (577h30m23.093368755s) [noise=true , address=45.91.116.10:4083 , piece_id=KWQNOQ6HQK3IBDYYZNHU6DKAIGUGI25S3CV4X3I5UO4LAQTSJPDQ, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.224011706 (577h30m23.095094742s) [noise=true , address=145.79.227.16:4909 , piece_id=FCSEDPTTSZ5PXBM4ANAZPOTHCAHBGQRYT3XLVOZF6YHEM3GUDKXQ, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.225944206 (577h30m23.097027251s) [noise=true , address=45.91.116.9:4263 , piece_id=PRVDFJE7K6PB2DHKQSVN5TFODABFPDLH7PBAUR64XUM2BPXUXOVQ, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:23.227790031 (577h30m23.098873066s) [noise=true , address=145.79.227.12:4765 , piece_id=Q7R42CIB7MQSOYJWKD6XSFQ7KBGRJCPKFZQWSULC5HS47OSDML6Q, seg_pos={0 0}, upload=506260]: Uploading piece... Dec 29 16:59:24.823206268 (577h30m24.694289313s) [upload=506260]: Next returned nil segment Dec 29 16:59:24.823230405 (577h30m24.69431345s) [upload=506260]: Waiting for error group managing segments... Dec 29 16:59:25.208464947 (577h30m25.079547982s) [noise=true , address=145.79.227.32:4836 , piece_id=AR27H3RUGPXEUURKFS4NQM354ZC4EKJVPVCRLRVVN7TVC452MZ4A, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.211452436 (577h30m25.082535471s) [noise=true , address=145.79.227.12:4765 , piece_id=Q7R42CIB7MQSOYJWKD6XSFQ7KBGRJCPKFZQWSULC5HS47OSDML6Q, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.216341546 (577h30m25.087424581s) [noise=true , address=145.79.227.48:4301 , piece_id=NSGQ4MHZLZ2V74BMHBWNYVOXGZOLWET2XXEDNU76JGKLPOFPQIYA, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.220586919 (577h30m25.091669954s) [noise=true , address=145.79.227.6:4257 , piece_id=AGTOP6RMQ4VHID6RRZ2JFRAHYVA5A3KGQ72YUUD5J6SOFNSHDAMQ, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.226752269 (577h30m25.097835314s) [noise=true , address=145.79.227.5:4221 , piece_id=GLUVY7N6KIKZY5MPGDGJK7RLUMCTZABTDZNMXKYUFFYWKNCZUM7Q, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.230025973 (577h30m25.101109008s) [noise=true , address=145.79.227.8:4329 , piece_id=DHT2VOOEJIYCIJMRUUO5MPDJDYMBWWFDMIVL7N4ANQQNEIBX6SBQ, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.237136714 (577h30m25.108219749s) [noise=true , address=145.79.227.47:4266 , piece_id=S76BYQ256IKABUXGWMCEI3PKEWEL3YWAS7NFSQGOPQWYGI3MPT7A, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.453059070 (577h30m25.324142105s) [noise=true , address=77.90.31.17:4773 , piece_id=SJMPVHE5NUABVLPQKKGN5FYD6LNOPRBSHA7UL6CP6A4ZHG6OAFDQ, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.463014257 (577h30m25.334097292s) [noise=true , address=45.91.116.9:4263 , piece_id=PRVDFJE7K6PB2DHKQSVN5TFODABFPDLH7PBAUR64XUM2BPXUXOVQ, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.465436580 (577h30m25.336519625s) [noise=true , address=77.90.31.15:4697 , piece_id=XSYVFBIDNX3OWGMJ2KMFFY6XMW5K7OGB5F6UBQXCNNJERSZBB7WA, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.494960227 (577h30m25.366043262s) [noise=true , address=77.90.31.16:4735 , piece_id=U72ALMBCMK3FV6JFQZ2BY7PGU5ET4ONMRS46LVDSFWKM2LGEZMTQ, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:25.534164207 (577h30m25.405247242s) [noise=true , address=45.91.116.12:4225 , piece_id=POF7HZELBAHMX4CD2WZACINZNOJ3L4ON6HATYYOSMVUBHORUJD3Q, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:26.163338718 (577h30m26.034421753s) [noise=true , address=45.91.116.10:4083 , piece_id=KWQNOQ6HQK3IBDYYZNHU6DKAIGUGI25S3CV4X3I5UO4LAQTSJPDQ, seg_pos={0 0}, upload=506260]: Done uploading piece. err: <nil> Dec 29 16:59:26.163362634 (577h30m26.034445669s) [seg_pos={0 0}, upload=506260]: Segment reached optimal threshold of 13 pieces. Dec 29 16:59:26.163551904 (577h30m26.034634939s) [noise=true , address=45.91.116.13:4155 , piece_id=LSEQKHCVE7RFLCTVOCL3OAVGINDPXMPH6IQBWFCG4UA5CONBUCNA, seg_pos={0 0}, upload=506260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:39032->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:39032->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39032->45.91.116.13:4155: use of closed network connection Dec 29 16:59:26.163557816 (577h30m26.034640851s) [noise=true , address=45.91.116.13:4155 , piece_id=LSEQKHCVE7RFLCTVOCL3OAVGINDPXMPH6IQBWFCG4UA5CONBUCNA, seg_pos={0 0}, upload=506260]: Long tail cancellation due to optimal threshold Dec 29 16:59:26.163740394 (577h30m26.034823429s) [noise=true , address=145.79.227.16:4909 , piece_id=FCSEDPTTSZ5PXBM4ANAZPOTHCAHBGQRYT3XLVOZF6YHEM3GUDKXQ, seg_pos={0 0}, upload=506260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:54214->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:54214->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54214->145.79.227.16:4909: use of closed network connection Dec 29 16:59:26.163748829 (577h30m26.034831865s) [noise=true , address=145.79.227.16:4909 , piece_id=FCSEDPTTSZ5PXBM4ANAZPOTHCAHBGQRYT3XLVOZF6YHEM3GUDKXQ, seg_pos={0 0}, upload=506260]: Long tail cancellation due to optimal threshold Dec 29 16:59:26.163841867 (577h30m26.034924902s) [noise=true , address=145.79.227.44:4194 , piece_id=RBO2B6FMLPW4K2IQWI4IR4ICDIJEEJO2ROSGUPSCKVA3PJLP2XJA, seg_pos={0 0}, upload=506260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 16:59:26.163888185 (577h30m26.03497122s) [noise=true , address=145.79.227.44:4194 , piece_id=RBO2B6FMLPW4K2IQWI4IR4ICDIJEEJO2ROSGUPSCKVA3PJLP2XJA, seg_pos={0 0}, upload=506260]: Long tail cancellation due to optimal threshold Dec 29 16:59:26.163919505 (577h30m26.03500254s) [seg_pos={0 0}, upload=506260]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:59:26.164082565 (577h30m26.0351656s) [upload=506260]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:59:26.164091141 (577h30m26.035174177s) [upload=506260]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:59:26.446085320 (577h30m26.317168395s) [upload=506260]: Done uploading segments... Dec 29 16:59:26.446114075 (577h30m26.31719711s) [upload=506260]: Upload finished. err: <nil> Duration: 30.002685799s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX.......] duration: 4.575966696s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 16:59:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.575991483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....] duration: 21.862955791s trace: https://jaeger.dev.storj.io/trace/152ea4763f32de53 FAIL (2025-12-29 16:59:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:56:06.543953773 (577h27m6.415036818s) [upload=506212]: Starting upload Dec 29 16:56:06.544049927 (577h27m6.415132972s) [upload=506212]: Done starting upload Dec 29 16:56:06.544194512 (577h27m6.415277557s) [upload=506212]: Uploading segments... Dec 29 16:56:06.544290214 (577h27m6.415373249s) [upload=506212]: Got next segment. Inline: false Dec 29 16:56:06.544349407 (577h27m6.415432452s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:56:06.544356911 (577h27m6.415439956s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:56:06.794548692 (577h27m6.665631738s) [seg_pos={0 0}, upload=506212]: Begin upload segment... Dec 29 16:56:06.794596153 (577h27m6.665679199s) [seg_pos={0 0}, upload=506212]: Done begin upload segment. Dec 29 16:56:06.794627843 (577h27m6.665710908s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.794716472 (577h27m6.665799527s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795345830 (577h27m6.666428865s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795823350 (577h27m6.666906395s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796271273 (577h27m6.667354308s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796732812 (577h27m6.667815857s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797283541 (577h27m6.668366576s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797902499 (577h27m6.668985544s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.798940806 (577h27m6.670023851s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.811455656 (577h27m6.682538691s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.812306276 (577h27m6.683389311s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.814683743 (577h27m6.685766848s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.815917783 (577h27m6.687000828s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.818154943 (577h27m6.689238048s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.819720875 (577h27m6.69080391s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.820145082 (577h27m6.691228127s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:08.442801480 (577h27m8.313884525s) [upload=506212]: Next returned nil segment Dec 29 16:56:08.442836517 (577h27m8.313919562s) [upload=506212]: Waiting for error group managing segments... Dec 29 16:56:08.783663392 (577h27m8.654746428s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.791128898 (577h27m8.662211943s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.797546287 (577h27m8.668629332s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.803391097 (577h27m8.674474132s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.817764727 (577h27m8.688847762s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.825946826 (577h27m8.697029871s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.829876899 (577h27m8.700959934s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.833292323 (577h27m8.704375358s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.835730736 (577h27m8.706813781s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.840300948 (577h27m8.711383983s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.023415566 (577h27m8.894498611s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.123044764 (577h27m8.994127809s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210678021 (577h27m9.081761066s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210701766 (577h27m9.081784811s) [seg_pos={0 0}, upload=506212]: Segment reached optimal threshold of 13 pieces. Dec 29 16:56:09.210970578 (577h27m9.082053613s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.210979555 (577h27m9.08206259s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211560301 (577h27m9.082643376s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.211588735 (577h27m9.08267177s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211733922 (577h27m9.082816977s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection Dec 29 16:56:09.211745664 (577h27m9.082828699s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211772665 (577h27m9.08285571s) [seg_pos={0 0}, upload=506212]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:56:09.211944252 (577h27m9.083027287s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:09.211951906 (577h27m9.083034941s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:09.513386052 (577h27m9.384469087s) [upload=506212]: Done uploading segments... Dec 29 16:56:09.513405479 (577h27m9.384488514s) [upload=506212]: Upload finished. err: <nil> Duration: 21.862980136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X......................X..] duration: 33.593426141s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 16:59:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 33.593469935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX......] duration: 5.000632402s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 16:58:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000657169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X...] duration: 24.000511464s trace: https://jaeger.dev.storj.io/trace/152ea4763f32de53 FAIL (2025-12-29 16:58:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:56:06.543953773 (577h27m6.415036818s) [upload=506212]: Starting upload Dec 29 16:56:06.544049927 (577h27m6.415132972s) [upload=506212]: Done starting upload Dec 29 16:56:06.544194512 (577h27m6.415277557s) [upload=506212]: Uploading segments... Dec 29 16:56:06.544290214 (577h27m6.415373249s) [upload=506212]: Got next segment. Inline: false Dec 29 16:56:06.544349407 (577h27m6.415432452s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:56:06.544356911 (577h27m6.415439956s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:56:06.794548692 (577h27m6.665631738s) [seg_pos={0 0}, upload=506212]: Begin upload segment... Dec 29 16:56:06.794596153 (577h27m6.665679199s) [seg_pos={0 0}, upload=506212]: Done begin upload segment. Dec 29 16:56:06.794627843 (577h27m6.665710908s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.794716472 (577h27m6.665799527s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795345830 (577h27m6.666428865s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795823350 (577h27m6.666906395s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796271273 (577h27m6.667354308s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796732812 (577h27m6.667815857s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797283541 (577h27m6.668366576s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797902499 (577h27m6.668985544s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.798940806 (577h27m6.670023851s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.811455656 (577h27m6.682538691s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.812306276 (577h27m6.683389311s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.814683743 (577h27m6.685766848s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.815917783 (577h27m6.687000828s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.818154943 (577h27m6.689238048s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.819720875 (577h27m6.69080391s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.820145082 (577h27m6.691228127s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:08.442801480 (577h27m8.313884525s) [upload=506212]: Next returned nil segment Dec 29 16:56:08.442836517 (577h27m8.313919562s) [upload=506212]: Waiting for error group managing segments... Dec 29 16:56:08.783663392 (577h27m8.654746428s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.791128898 (577h27m8.662211943s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.797546287 (577h27m8.668629332s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.803391097 (577h27m8.674474132s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.817764727 (577h27m8.688847762s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.825946826 (577h27m8.697029871s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.829876899 (577h27m8.700959934s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.833292323 (577h27m8.704375358s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.835730736 (577h27m8.706813781s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.840300948 (577h27m8.711383983s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.023415566 (577h27m8.894498611s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.123044764 (577h27m8.994127809s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210678021 (577h27m9.081761066s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210701766 (577h27m9.081784811s) [seg_pos={0 0}, upload=506212]: Segment reached optimal threshold of 13 pieces. Dec 29 16:56:09.210970578 (577h27m9.082053613s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.210979555 (577h27m9.08206259s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211560301 (577h27m9.082643376s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.211588735 (577h27m9.08267177s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211733922 (577h27m9.082816977s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection Dec 29 16:56:09.211745664 (577h27m9.082828699s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211772665 (577h27m9.08285571s) [seg_pos={0 0}, upload=506212]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:56:09.211944252 (577h27m9.083027287s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:09.211951906 (577h27m9.083034941s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:09.513386052 (577h27m9.384469087s) [upload=506212]: Done uploading segments... Dec 29 16:56:09.513405479 (577h27m9.384488514s) [upload=506212]: Upload finished. err: <nil> Duration: 24.000540679s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX.....] duration: 4.684927184s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 16:57:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.684953805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X......................X.] duration: 22.917720562s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 16:57:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 22.9177405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X..] duration: 5.998475472s trace: https://jaeger.dev.storj.io/trace/152ea4763f32de53 FAIL (2025-12-29 16:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:56:06.543953773 (577h27m6.415036818s) [upload=506212]: Starting upload Dec 29 16:56:06.544049927 (577h27m6.415132972s) [upload=506212]: Done starting upload Dec 29 16:56:06.544194512 (577h27m6.415277557s) [upload=506212]: Uploading segments... Dec 29 16:56:06.544290214 (577h27m6.415373249s) [upload=506212]: Got next segment. Inline: false Dec 29 16:56:06.544349407 (577h27m6.415432452s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:56:06.544356911 (577h27m6.415439956s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:56:06.794548692 (577h27m6.665631738s) [seg_pos={0 0}, upload=506212]: Begin upload segment... Dec 29 16:56:06.794596153 (577h27m6.665679199s) [seg_pos={0 0}, upload=506212]: Done begin upload segment. Dec 29 16:56:06.794627843 (577h27m6.665710908s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.794716472 (577h27m6.665799527s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795345830 (577h27m6.666428865s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795823350 (577h27m6.666906395s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796271273 (577h27m6.667354308s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796732812 (577h27m6.667815857s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797283541 (577h27m6.668366576s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797902499 (577h27m6.668985544s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.798940806 (577h27m6.670023851s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.811455656 (577h27m6.682538691s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.812306276 (577h27m6.683389311s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.814683743 (577h27m6.685766848s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.815917783 (577h27m6.687000828s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.818154943 (577h27m6.689238048s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.819720875 (577h27m6.69080391s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.820145082 (577h27m6.691228127s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:08.442801480 (577h27m8.313884525s) [upload=506212]: Next returned nil segment Dec 29 16:56:08.442836517 (577h27m8.313919562s) [upload=506212]: Waiting for error group managing segments... Dec 29 16:56:08.783663392 (577h27m8.654746428s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.791128898 (577h27m8.662211943s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.797546287 (577h27m8.668629332s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.803391097 (577h27m8.674474132s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.817764727 (577h27m8.688847762s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.825946826 (577h27m8.697029871s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.829876899 (577h27m8.700959934s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.833292323 (577h27m8.704375358s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.835730736 (577h27m8.706813781s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.840300948 (577h27m8.711383983s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.023415566 (577h27m8.894498611s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.123044764 (577h27m8.994127809s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210678021 (577h27m9.081761066s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210701766 (577h27m9.081784811s) [seg_pos={0 0}, upload=506212]: Segment reached optimal threshold of 13 pieces. Dec 29 16:56:09.210970578 (577h27m9.082053613s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.210979555 (577h27m9.08206259s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211560301 (577h27m9.082643376s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.211588735 (577h27m9.08267177s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211733922 (577h27m9.082816977s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection Dec 29 16:56:09.211745664 (577h27m9.082828699s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211772665 (577h27m9.08285571s) [seg_pos={0 0}, upload=506212]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:56:09.211944252 (577h27m9.083027287s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:09.211951906 (577h27m9.083034941s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:09.513386052 (577h27m9.384469087s) [upload=506212]: Done uploading segments... Dec 29 16:56:09.513405479 (577h27m9.384488514s) [upload=506212]: Upload finished. err: <nil> Duration: 5.99849991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......X......................X] duration: 1m0.005897672s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 16:57:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 1m0.005924995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX....] duration: 5.00122959s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 16:57:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001248466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X.] duration: 6.18691127s trace: https://jaeger.dev.storj.io/trace/152ea4763f32de53 FAIL (2025-12-29 16:56:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:56:06.543953773 (577h27m6.415036818s) [upload=506212]: Starting upload Dec 29 16:56:06.544049927 (577h27m6.415132972s) [upload=506212]: Done starting upload Dec 29 16:56:06.544194512 (577h27m6.415277557s) [upload=506212]: Uploading segments... Dec 29 16:56:06.544290214 (577h27m6.415373249s) [upload=506212]: Got next segment. Inline: false Dec 29 16:56:06.544349407 (577h27m6.415432452s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:56:06.544356911 (577h27m6.415439956s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:56:06.794548692 (577h27m6.665631738s) [seg_pos={0 0}, upload=506212]: Begin upload segment... Dec 29 16:56:06.794596153 (577h27m6.665679199s) [seg_pos={0 0}, upload=506212]: Done begin upload segment. Dec 29 16:56:06.794627843 (577h27m6.665710908s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.794716472 (577h27m6.665799527s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795345830 (577h27m6.666428865s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795823350 (577h27m6.666906395s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796271273 (577h27m6.667354308s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796732812 (577h27m6.667815857s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797283541 (577h27m6.668366576s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797902499 (577h27m6.668985544s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.798940806 (577h27m6.670023851s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.811455656 (577h27m6.682538691s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.812306276 (577h27m6.683389311s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.814683743 (577h27m6.685766848s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.815917783 (577h27m6.687000828s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.818154943 (577h27m6.689238048s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.819720875 (577h27m6.69080391s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.820145082 (577h27m6.691228127s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:08.442801480 (577h27m8.313884525s) [upload=506212]: Next returned nil segment Dec 29 16:56:08.442836517 (577h27m8.313919562s) [upload=506212]: Waiting for error group managing segments... Dec 29 16:56:08.783663392 (577h27m8.654746428s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.791128898 (577h27m8.662211943s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.797546287 (577h27m8.668629332s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.803391097 (577h27m8.674474132s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.817764727 (577h27m8.688847762s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.825946826 (577h27m8.697029871s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.829876899 (577h27m8.700959934s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.833292323 (577h27m8.704375358s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.835730736 (577h27m8.706813781s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.840300948 (577h27m8.711383983s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.023415566 (577h27m8.894498611s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.123044764 (577h27m8.994127809s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210678021 (577h27m9.081761066s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210701766 (577h27m9.081784811s) [seg_pos={0 0}, upload=506212]: Segment reached optimal threshold of 13 pieces. Dec 29 16:56:09.210970578 (577h27m9.082053613s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.210979555 (577h27m9.08206259s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211560301 (577h27m9.082643376s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.211588735 (577h27m9.08267177s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211733922 (577h27m9.082816977s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection Dec 29 16:56:09.211745664 (577h27m9.082828699s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211772665 (577h27m9.08285571s) [seg_pos={0 0}, upload=506212]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:56:09.211944252 (577h27m9.083027287s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:09.211951906 (577h27m9.083034941s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:09.513386052 (577h27m9.384469087s) [upload=506212]: Done uploading segments... Dec 29 16:56:09.513405479 (577h27m9.384488514s) [upload=506212]: Upload finished. err: <nil> Duration: 6.1869384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [..X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X] duration: 30.002681867s trace: https://jaeger.dev.storj.io/trace/152ea4763f32de53 FAIL (2025-12-29 16:56:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:56:06.543953773 (577h27m6.415036818s) [upload=506212]: Starting upload Dec 29 16:56:06.544049927 (577h27m6.415132972s) [upload=506212]: Done starting upload Dec 29 16:56:06.544194512 (577h27m6.415277557s) [upload=506212]: Uploading segments... Dec 29 16:56:06.544290214 (577h27m6.415373249s) [upload=506212]: Got next segment. Inline: false Dec 29 16:56:06.544349407 (577h27m6.415432452s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:56:06.544356911 (577h27m6.415439956s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:56:06.794548692 (577h27m6.665631738s) [seg_pos={0 0}, upload=506212]: Begin upload segment... Dec 29 16:56:06.794596153 (577h27m6.665679199s) [seg_pos={0 0}, upload=506212]: Done begin upload segment. Dec 29 16:56:06.794627843 (577h27m6.665710908s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.794716472 (577h27m6.665799527s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795345830 (577h27m6.666428865s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.795823350 (577h27m6.666906395s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796271273 (577h27m6.667354308s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.796732812 (577h27m6.667815857s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797283541 (577h27m6.668366576s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.797902499 (577h27m6.668985544s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.798940806 (577h27m6.670023851s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.811455656 (577h27m6.682538691s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.812306276 (577h27m6.683389311s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.814683743 (577h27m6.685766848s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.815917783 (577h27m6.687000828s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.818154943 (577h27m6.689238048s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.819720875 (577h27m6.69080391s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:06.820145082 (577h27m6.691228127s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Uploading piece... Dec 29 16:56:08.442801480 (577h27m8.313884525s) [upload=506212]: Next returned nil segment Dec 29 16:56:08.442836517 (577h27m8.313919562s) [upload=506212]: Waiting for error group managing segments... Dec 29 16:56:08.783663392 (577h27m8.654746428s) [noise=true , address=145.79.227.6:4258 , piece_id=UNOZ5SPM46HJ27TQPNUKAO543FDDVYB7FY7WALD4M63AUCMJJH5A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.791128898 (577h27m8.662211943s) [noise=true , address=145.79.227.48:4302 , piece_id=QZYLQLMMIJ2SPF2WZDFCRSYMEHSGPYFVD5SAC5AWHSUEGOE4TPNQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.797546287 (577h27m8.668629332s) [noise=true , address=145.79.227.10:4403 , piece_id=4FUYKV5ZV4CKEQ7ZQADEHRKH6ZLJ2W4VV3433AOS26E3EH3GCEKQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.803391097 (577h27m8.674474132s) [noise=true , address=145.79.227.45:4229 , piece_id=VZDCUNKVULEYVAUN5EBZMARFOVE4ZCYIT34STS6BL6JGCYNYU5CA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.817764727 (577h27m8.688847762s) [noise=true , address=145.79.227.28:4725 , piece_id=EYDOFUW22BD36BJYJ4I2AI5XJGDGR7GWHV3HERX2O62N6RXOQEAA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.825946826 (577h27m8.697029871s) [noise=true , address=145.79.227.11:4729 , piece_id=GM2JYLLVD6WURXX5UMKNYSO4U45NLAJXRXBE6JD7WSRE4KQP22QQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.829876899 (577h27m8.700959934s) [noise=true , address=145.79.227.37:4567 , piece_id=SD7GHV64SQBPPWN5PWMLLQK2QIZ4JLWB4EEGRVBKDP25REF7QWBQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.833292323 (577h27m8.704375358s) [noise=true , address=145.79.227.7:4295 , piece_id=ZVGBQWCJSUJWOLJW7E42ZONU26KAFVZVCURGMW6ECWHM7OVUOAHA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.835730736 (577h27m8.706813781s) [noise=true , address=145.79.227.47:4266 , piece_id=HBCF3GF25I3TEMEYJHYC2433RTAFN5MV62BWKHBCTP4DDWS4MK3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:08.840300948 (577h27m8.711383983s) [noise=true , address=145.79.227.42:4122 , piece_id=FAOTLS53NGNL6OCR2KMA2RMRTWNHGFFFWPMUN2PVUURPAISDE4LA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.023415566 (577h27m8.894498611s) [noise=true , address=77.90.31.12:4590 , piece_id=GWB6DTARFSCQYTFN4QXIAYXFIX37K6AYCF42XWQZ74HRDLBKKLKA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.123044764 (577h27m8.994127809s) [noise=true , address=77.90.31.21:4878 , piece_id=MLXFNBKADEUBQXWS4CZPN57EM7PWVK7H4TRN2EHMPS5VVAA52BOA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210678021 (577h27m9.081761066s) [noise=true , address=45.91.116.8:4045 , piece_id=TKLDR5U2NPMB4CD75FIANTXQDKCRO5MWUXGPVPX2HTOM6HCRAKQA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: <nil> Dec 29 16:56:09.210701766 (577h27m9.081784811s) [seg_pos={0 0}, upload=506212]: Segment reached optimal threshold of 13 pieces. Dec 29 16:56:09.210970578 (577h27m9.082053613s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.210979555 (577h27m9.08206259s) [noise=true , address=45.91.116.4:4939 , piece_id=7UGYSBPTRUA3IMLDZ7UB3IUTKYHN5XHHSO4UOLOMIWIGS3GDYSPA, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211560301 (577h27m9.082643376s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:09.211588735 (577h27m9.08267177s) [noise=true , address=45.91.116.11:4117 , piece_id=MJTQHC6K6BXDBFI3ZA666WO2SFGHZQMCBIGJPVBWNRQKU4IAOP3A, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211733922 (577h27m9.082816977s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56500->145.79.227.2:4113: use of closed network connection Dec 29 16:56:09.211745664 (577h27m9.082828699s) [noise=true , address=145.79.227.2:4113 , piece_id=XKXR72DU7ZW5B4JIEXYBZMJV4PNXQMIZJZ57QENO6SHV7MVFAVZQ, seg_pos={0 0}, upload=506212]: Long tail cancellation due to optimal threshold Dec 29 16:56:09.211772665 (577h27m9.08285571s) [seg_pos={0 0}, upload=506212]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:56:09.211944252 (577h27m9.083027287s) [upload=506212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:09.211951906 (577h27m9.083034941s) [upload=506212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:09.513386052 (577h27m9.384469087s) [upload=506212]: Done uploading segments... Dec 29 16:56:09.513405479 (577h27m9.384488514s) [upload=506212]: Upload finished. err: <nil> Duration: 30.00270403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX...] duration: 4.957777815s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 16:56:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.957825998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.......X......................] duration: 30.507970516s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:56:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 30.507994613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.......X.....................] duration: 38.874991463s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:55:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 38.875007202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX..] duration: 4.957975839s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 16:55:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.958004574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.......X....................] duration: 36.329862339s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 36.329893048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [...X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.] duration: 5.753165366s trace: https://jaeger.dev.storj.io/trace/7e74c817b902ad49 FAIL (2025-12-29 16:55:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: LKM7FV3H6RI6JGQTK6HETHJ5R423CNWVDW2B2C7ZCQTXMSHBHQ4A) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: YEALSD454JJ6EVELZZCFRC46HC4BLROJB5ZETJRP54LHBBT7FSYA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: LFVKT7MQHQQJULQUL3H5N4JV6RB5DRKT4Z4P2YCEYQRHTRTWQBDQ) context deadline exceeded ======LOGS====== Dec 29 16:54:24.754349478 (577h25m24.625432513s) [upload=506187]: Starting upload Dec 29 16:54:24.754419531 (577h25m24.625502576s) [upload=506187]: Done starting upload Dec 29 16:54:24.754562184 (577h25m24.625645219s) [upload=506187]: Uploading segments... Dec 29 16:54:24.754623550 (577h25m24.625706585s) [upload=506187]: Got next segment. Inline: false Dec 29 16:54:24.754670900 (577h25m24.625753936s) [upload=506187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:54:24.754679106 (577h25m24.625762141s) [upload=506187]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:54:24.999780253 (577h25m24.870863298s) [seg_pos={0 0}, upload=506187]: Begin upload segment... Dec 29 16:54:24.999843994 (577h25m24.870927029s) [seg_pos={0 0}, upload=506187]: Done begin upload segment. Dec 29 16:54:24.999875805 (577h25m24.87095884s) [noise=true , address=45.91.116.13:4153 , piece_id=R42OBU2JDEAJTRLEFCAUY3R5I3O6PPFGRYUCQXZ4CKONES72I3FQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.000044426 (577h25m24.871127491s) [noise=true , address=45.91.116.12:4227 , piece_id=SYQAPJ4YNKP7LMJC47772XNLZK5IKEZHPNUAMY3VOTGZV5O6SSDA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.000650099 (577h25m24.871733134s) [noise=true , address=77.90.31.14:4662 , piece_id=GYNZZORXBYNQUZKUF5YFI5X4R3DFNMVPP4MFVH354FM333V27CAQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.001166743 (577h25m24.872249788s) [noise=true , address=77.90.31.12:4590 , piece_id=YEALSD454JJ6EVELZZCFRC46HC4BLROJB5ZETJRP54LHBBT7FSYA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.001565192 (577h25m24.872648237s) [noise=true , address=145.79.227.50:4375 , piece_id=UDV5IACQZNOWNPSPQMPIWNCC3S3O2CFYEQ2NAAM43BOB3D5RTY5A, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.001986344 (577h25m24.873069379s) [noise=true , address=45.91.116.9:4263 , piece_id=LFVKT7MQHQQJULQUL3H5N4JV6RB5DRKT4Z4P2YCEYQRHTRTWQBDQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.001999459 (577h25m24.873082504s) [noise=true , address=45.91.116.11:4117 , piece_id=54654JLPYKFLAFHITUX6FPZZROY72EYKTJ4GSZWQZI5Q2LPVW3MA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.002395613 (577h25m24.873478648s) [noise=true , address=145.79.227.47:4267 , piece_id=6ZX7DGUYLWN6UC4QKH24ANJSONNDGX62GAVP3K37JNR3L4ZM6OZA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.002452121 (577h25m24.873535166s) [noise=true , address=145.79.227.34:4907 , piece_id=YJMGZ437Q5DWV4U4M2K77U6QWGY3DXQRP7MCTKP3YHVVHE7AWR2A, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.002782200 (577h25m24.873865245s) [noise=true , address=77.90.31.21:4877 , piece_id=DWDOIFL5S77MSEHIOVOFMSO7Y37GQU2NLWKJ7GOH4SNVKVQM2JWQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.002928137 (577h25m24.874011173s) [noise=true , address=145.79.227.8:4329 , piece_id=ONDYYVU442HBTCOUHTWG6UVS73DCWM2VRCJMR6DNQWZ6JKTPKJNA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.003182742 (577h25m24.874265787s) [noise=true , address=45.91.116.4:4939 , piece_id=AMTYSKRYADON5YTFJZK7HYBECZ65WW7WHK6JTING3ZZG52BHPVBQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.003413882 (577h25m24.874496928s) [noise=true , address=145.79.227.16:4909 , piece_id=Z4TD2QL4N2WOSW3NHHZ56WMVSLF5GPFSZIG4LY23YS75BIJIJMWQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.003680790 (577h25m24.874763825s) [noise=true , address=45.91.116.10:4084 , piece_id=LKM7FV3H6RI6JGQTK6HETHJ5R423CNWVDW2B2C7ZCQTXMSHBHQ4A, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.003930917 (577h25m24.875013952s) [noise=true , address=145.79.227.23:4546 , piece_id=M3BBLRZ5NE5MOUK35S7K4OVCEKQR3K7DHOOZ6YQLBC4SJXXKUGIQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.004202964 (577h25m24.875286009s) [noise=true , address=45.91.116.5:4299 , piece_id=GECMY3GVB4N6ZCYAS22INJNGI36OCDJ5RZ3V7VSNJJEB7S7EEDEA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:26.706849054 (577h25m26.577932099s) [upload=506187]: Next returned nil segment Dec 29 16:54:26.706864713 (577h25m26.577947758s) [upload=506187]: Waiting for error group managing segments... Dec 29 16:54:26.988192343 (577h25m26.859275378s) [noise=true , address=145.79.227.8:4329 , piece_id=ONDYYVU442HBTCOUHTWG6UVS73DCWM2VRCJMR6DNQWZ6JKTPKJNA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.001944700 (577h25m26.873027765s) [noise=true , address=145.79.227.50:4375 , piece_id=UDV5IACQZNOWNPSPQMPIWNCC3S3O2CFYEQ2NAAM43BOB3D5RTY5A, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.022376204 (577h25m26.893459249s) [noise=true , address=145.79.227.16:4909 , piece_id=Z4TD2QL4N2WOSW3NHHZ56WMVSLF5GPFSZIG4LY23YS75BIJIJMWQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.028813562 (577h25m26.899896597s) [noise=true , address=145.79.227.47:4267 , piece_id=6ZX7DGUYLWN6UC4QKH24ANJSONNDGX62GAVP3K37JNR3L4ZM6OZA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.073851761 (577h25m26.944934806s) [noise=true , address=145.79.227.23:4546 , piece_id=M3BBLRZ5NE5MOUK35S7K4OVCEKQR3K7DHOOZ6YQLBC4SJXXKUGIQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.174984262 (577h25m27.046067307s) [noise=true , address=77.90.31.14:4662 , piece_id=GYNZZORXBYNQUZKUF5YFI5X4R3DFNMVPP4MFVH354FM333V27CAQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.218260386 (577h25m27.089343421s) [noise=true , address=77.90.31.12:4590 , piece_id=YEALSD454JJ6EVELZZCFRC46HC4BLROJB5ZETJRP54LHBBT7FSYA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.264881591 (577h25m27.135964626s) [noise=true , address=45.91.116.5:4299 , piece_id=GECMY3GVB4N6ZCYAS22INJNGI36OCDJ5RZ3V7VSNJJEB7S7EEDEA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.308266071 (577h25m27.179349106s) [noise=true , address=45.91.116.4:4939 , piece_id=AMTYSKRYADON5YTFJZK7HYBECZ65WW7WHK6JTING3ZZG52BHPVBQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.326604340 (577h25m27.197687375s) [noise=true , address=45.91.116.9:4263 , piece_id=LFVKT7MQHQQJULQUL3H5N4JV6RB5DRKT4Z4P2YCEYQRHTRTWQBDQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.341345749 (577h25m27.212428784s) [noise=true , address=45.91.116.10:4084 , piece_id=LKM7FV3H6RI6JGQTK6HETHJ5R423CNWVDW2B2C7ZCQTXMSHBHQ4A, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.361728451 (577h25m27.232811486s) [noise=true , address=45.91.116.13:4153 , piece_id=R42OBU2JDEAJTRLEFCAUY3R5I3O6PPFGRYUCQXZ4CKONES72I3FQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.943382395 (577h25m27.81446543s) [noise=true , address=45.91.116.12:4227 , piece_id=SYQAPJ4YNKP7LMJC47772XNLZK5IKEZHPNUAMY3VOTGZV5O6SSDA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.943405029 (577h25m27.814488074s) [seg_pos={0 0}, upload=506187]: Segment reached optimal threshold of 13 pieces. Dec 29 16:54:27.943602134 (577h25m27.814685179s) [noise=true , address=45.91.116.11:4117 , piece_id=54654JLPYKFLAFHITUX6FPZZROY72EYKTJ4GSZWQZI5Q2LPVW3MA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 16:54:27.943608946 (577h25m27.814691981s) [noise=true , address=45.91.116.11:4117 , piece_id=54654JLPYKFLAFHITUX6FPZZROY72EYKTJ4GSZWQZI5Q2LPVW3MA, seg_pos={0 0}, upload=506187]: Long tail cancellation due to optimal threshold Dec 29 16:54:27.943811432 (577h25m27.814894467s) [noise=true , address=145.79.227.34:4907 , piece_id=YJMGZ437Q5DWV4U4M2K77U6QWGY3DXQRP7MCTKP3YHVVHE7AWR2A, seg_pos={0 0}, upload=506187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 16:54:27.943817944 (577h25m27.814900979s) [noise=true , address=145.79.227.34:4907 , piece_id=YJMGZ437Q5DWV4U4M2K77U6QWGY3DXQRP7MCTKP3YHVVHE7AWR2A, seg_pos={0 0}, upload=506187]: Long tail cancellation due to optimal threshold Dec 29 16:54:27.943849334 (577h25m27.814932379s) [noise=true , address=77.90.31.21:4877 , piece_id=DWDOIFL5S77MSEHIOVOFMSO7Y37GQU2NLWKJ7GOH4SNVKVQM2JWQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 16:54:27.943879462 (577h25m27.814962497s) [noise=true , address=77.90.31.21:4877 , piece_id=DWDOIFL5S77MSEHIOVOFMSO7Y37GQU2NLWKJ7GOH4SNVKVQM2JWQ, seg_pos={0 0}, upload=506187]: Long tail cancellation due to optimal threshold Dec 29 16:54:27.943899831 (577h25m27.814982876s) [seg_pos={0 0}, upload=506187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:54:27.943990503 (577h25m27.815073538s) [upload=506187]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:54:27.943995523 (577h25m27.815078568s) [upload=506187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:54:28.232525557 (577h25m28.103608612s) [upload=506187]: Done uploading segments... Dec 29 16:54:28.232547729 (577h25m28.103630774s) [upload=506187]: Upload finished. err: <nil> Duration: 5.753180404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [....X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX] duration: 30.00262673s trace: https://jaeger.dev.storj.io/trace/7e74c817b902ad49 FAIL (2025-12-29 16:54:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: LKM7FV3H6RI6JGQTK6HETHJ5R423CNWVDW2B2C7ZCQTXMSHBHQ4A) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: YEALSD454JJ6EVELZZCFRC46HC4BLROJB5ZETJRP54LHBBT7FSYA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: LFVKT7MQHQQJULQUL3H5N4JV6RB5DRKT4Z4P2YCEYQRHTRTWQBDQ) context deadline exceeded ======LOGS====== Dec 29 16:54:24.754349478 (577h25m24.625432513s) [upload=506187]: Starting upload Dec 29 16:54:24.754419531 (577h25m24.625502576s) [upload=506187]: Done starting upload Dec 29 16:54:24.754562184 (577h25m24.625645219s) [upload=506187]: Uploading segments... Dec 29 16:54:24.754623550 (577h25m24.625706585s) [upload=506187]: Got next segment. Inline: false Dec 29 16:54:24.754670900 (577h25m24.625753936s) [upload=506187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:54:24.754679106 (577h25m24.625762141s) [upload=506187]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:54:24.999780253 (577h25m24.870863298s) [seg_pos={0 0}, upload=506187]: Begin upload segment... Dec 29 16:54:24.999843994 (577h25m24.870927029s) [seg_pos={0 0}, upload=506187]: Done begin upload segment. Dec 29 16:54:24.999875805 (577h25m24.87095884s) [noise=true , address=45.91.116.13:4153 , piece_id=R42OBU2JDEAJTRLEFCAUY3R5I3O6PPFGRYUCQXZ4CKONES72I3FQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.000044426 (577h25m24.871127491s) [noise=true , address=45.91.116.12:4227 , piece_id=SYQAPJ4YNKP7LMJC47772XNLZK5IKEZHPNUAMY3VOTGZV5O6SSDA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.000650099 (577h25m24.871733134s) [noise=true , address=77.90.31.14:4662 , piece_id=GYNZZORXBYNQUZKUF5YFI5X4R3DFNMVPP4MFVH354FM333V27CAQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.001166743 (577h25m24.872249788s) [noise=true , address=77.90.31.12:4590 , piece_id=YEALSD454JJ6EVELZZCFRC46HC4BLROJB5ZETJRP54LHBBT7FSYA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.001565192 (577h25m24.872648237s) [noise=true , address=145.79.227.50:4375 , piece_id=UDV5IACQZNOWNPSPQMPIWNCC3S3O2CFYEQ2NAAM43BOB3D5RTY5A, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.001986344 (577h25m24.873069379s) [noise=true , address=45.91.116.9:4263 , piece_id=LFVKT7MQHQQJULQUL3H5N4JV6RB5DRKT4Z4P2YCEYQRHTRTWQBDQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.001999459 (577h25m24.873082504s) [noise=true , address=45.91.116.11:4117 , piece_id=54654JLPYKFLAFHITUX6FPZZROY72EYKTJ4GSZWQZI5Q2LPVW3MA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.002395613 (577h25m24.873478648s) [noise=true , address=145.79.227.47:4267 , piece_id=6ZX7DGUYLWN6UC4QKH24ANJSONNDGX62GAVP3K37JNR3L4ZM6OZA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.002452121 (577h25m24.873535166s) [noise=true , address=145.79.227.34:4907 , piece_id=YJMGZ437Q5DWV4U4M2K77U6QWGY3DXQRP7MCTKP3YHVVHE7AWR2A, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.002782200 (577h25m24.873865245s) [noise=true , address=77.90.31.21:4877 , piece_id=DWDOIFL5S77MSEHIOVOFMSO7Y37GQU2NLWKJ7GOH4SNVKVQM2JWQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.002928137 (577h25m24.874011173s) [noise=true , address=145.79.227.8:4329 , piece_id=ONDYYVU442HBTCOUHTWG6UVS73DCWM2VRCJMR6DNQWZ6JKTPKJNA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.003182742 (577h25m24.874265787s) [noise=true , address=45.91.116.4:4939 , piece_id=AMTYSKRYADON5YTFJZK7HYBECZ65WW7WHK6JTING3ZZG52BHPVBQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.003413882 (577h25m24.874496928s) [noise=true , address=145.79.227.16:4909 , piece_id=Z4TD2QL4N2WOSW3NHHZ56WMVSLF5GPFSZIG4LY23YS75BIJIJMWQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.003680790 (577h25m24.874763825s) [noise=true , address=45.91.116.10:4084 , piece_id=LKM7FV3H6RI6JGQTK6HETHJ5R423CNWVDW2B2C7ZCQTXMSHBHQ4A, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.003930917 (577h25m24.875013952s) [noise=true , address=145.79.227.23:4546 , piece_id=M3BBLRZ5NE5MOUK35S7K4OVCEKQR3K7DHOOZ6YQLBC4SJXXKUGIQ, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:25.004202964 (577h25m24.875286009s) [noise=true , address=45.91.116.5:4299 , piece_id=GECMY3GVB4N6ZCYAS22INJNGI36OCDJ5RZ3V7VSNJJEB7S7EEDEA, seg_pos={0 0}, upload=506187]: Uploading piece... Dec 29 16:54:26.706849054 (577h25m26.577932099s) [upload=506187]: Next returned nil segment Dec 29 16:54:26.706864713 (577h25m26.577947758s) [upload=506187]: Waiting for error group managing segments... Dec 29 16:54:26.988192343 (577h25m26.859275378s) [noise=true , address=145.79.227.8:4329 , piece_id=ONDYYVU442HBTCOUHTWG6UVS73DCWM2VRCJMR6DNQWZ6JKTPKJNA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.001944700 (577h25m26.873027765s) [noise=true , address=145.79.227.50:4375 , piece_id=UDV5IACQZNOWNPSPQMPIWNCC3S3O2CFYEQ2NAAM43BOB3D5RTY5A, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.022376204 (577h25m26.893459249s) [noise=true , address=145.79.227.16:4909 , piece_id=Z4TD2QL4N2WOSW3NHHZ56WMVSLF5GPFSZIG4LY23YS75BIJIJMWQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.028813562 (577h25m26.899896597s) [noise=true , address=145.79.227.47:4267 , piece_id=6ZX7DGUYLWN6UC4QKH24ANJSONNDGX62GAVP3K37JNR3L4ZM6OZA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.073851761 (577h25m26.944934806s) [noise=true , address=145.79.227.23:4546 , piece_id=M3BBLRZ5NE5MOUK35S7K4OVCEKQR3K7DHOOZ6YQLBC4SJXXKUGIQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.174984262 (577h25m27.046067307s) [noise=true , address=77.90.31.14:4662 , piece_id=GYNZZORXBYNQUZKUF5YFI5X4R3DFNMVPP4MFVH354FM333V27CAQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.218260386 (577h25m27.089343421s) [noise=true , address=77.90.31.12:4590 , piece_id=YEALSD454JJ6EVELZZCFRC46HC4BLROJB5ZETJRP54LHBBT7FSYA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.264881591 (577h25m27.135964626s) [noise=true , address=45.91.116.5:4299 , piece_id=GECMY3GVB4N6ZCYAS22INJNGI36OCDJ5RZ3V7VSNJJEB7S7EEDEA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.308266071 (577h25m27.179349106s) [noise=true , address=45.91.116.4:4939 , piece_id=AMTYSKRYADON5YTFJZK7HYBECZ65WW7WHK6JTING3ZZG52BHPVBQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.326604340 (577h25m27.197687375s) [noise=true , address=45.91.116.9:4263 , piece_id=LFVKT7MQHQQJULQUL3H5N4JV6RB5DRKT4Z4P2YCEYQRHTRTWQBDQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.341345749 (577h25m27.212428784s) [noise=true , address=45.91.116.10:4084 , piece_id=LKM7FV3H6RI6JGQTK6HETHJ5R423CNWVDW2B2C7ZCQTXMSHBHQ4A, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.361728451 (577h25m27.232811486s) [noise=true , address=45.91.116.13:4153 , piece_id=R42OBU2JDEAJTRLEFCAUY3R5I3O6PPFGRYUCQXZ4CKONES72I3FQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.943382395 (577h25m27.81446543s) [noise=true , address=45.91.116.12:4227 , piece_id=SYQAPJ4YNKP7LMJC47772XNLZK5IKEZHPNUAMY3VOTGZV5O6SSDA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: <nil> Dec 29 16:54:27.943405029 (577h25m27.814488074s) [seg_pos={0 0}, upload=506187]: Segment reached optimal threshold of 13 pieces. Dec 29 16:54:27.943602134 (577h25m27.814685179s) [noise=true , address=45.91.116.11:4117 , piece_id=54654JLPYKFLAFHITUX6FPZZROY72EYKTJ4GSZWQZI5Q2LPVW3MA, seg_pos={0 0}, upload=506187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 16:54:27.943608946 (577h25m27.814691981s) [noise=true , address=45.91.116.11:4117 , piece_id=54654JLPYKFLAFHITUX6FPZZROY72EYKTJ4GSZWQZI5Q2LPVW3MA, seg_pos={0 0}, upload=506187]: Long tail cancellation due to optimal threshold Dec 29 16:54:27.943811432 (577h25m27.814894467s) [noise=true , address=145.79.227.34:4907 , piece_id=YJMGZ437Q5DWV4U4M2K77U6QWGY3DXQRP7MCTKP3YHVVHE7AWR2A, seg_pos={0 0}, upload=506187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 16:54:27.943817944 (577h25m27.814900979s) [noise=true , address=145.79.227.34:4907 , piece_id=YJMGZ437Q5DWV4U4M2K77U6QWGY3DXQRP7MCTKP3YHVVHE7AWR2A, seg_pos={0 0}, upload=506187]: Long tail cancellation due to optimal threshold Dec 29 16:54:27.943849334 (577h25m27.814932379s) [noise=true , address=77.90.31.21:4877 , piece_id=DWDOIFL5S77MSEHIOVOFMSO7Y37GQU2NLWKJ7GOH4SNVKVQM2JWQ, seg_pos={0 0}, upload=506187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 16:54:27.943879462 (577h25m27.814962497s) [noise=true , address=77.90.31.21:4877 , piece_id=DWDOIFL5S77MSEHIOVOFMSO7Y37GQU2NLWKJ7GOH4SNVKVQM2JWQ, seg_pos={0 0}, upload=506187]: Long tail cancellation due to optimal threshold Dec 29 16:54:27.943899831 (577h25m27.814982876s) [seg_pos={0 0}, upload=506187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:54:27.943990503 (577h25m27.815073538s) [upload=506187]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:54:27.943995523 (577h25m27.815078568s) [upload=506187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:54:28.232525557 (577h25m28.103608612s) [upload=506187]: Done uploading segments... Dec 29 16:54:28.232547729 (577h25m28.103630774s) [upload=506187]: Upload finished. err: <nil> Duration: 30.002652198s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX.] duration: 4.432279685s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 16:54:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.43236182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.......X...................] duration: 51.802265124s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:54:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 51.802279932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.....X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.X] duration: 30.003345404s trace: https://jaeger.dev.storj.io/trace/25522584847307c5 FAIL (2025-12-29 16:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2GCBYQ4EQV46G6NQME2QO2HJP62YDXUL3LBAI4MAQQMAAOFFST5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MPZIMNCWWBBVS4CJVDBDVG5VSE2V42674TQG2BSGVC4PCGP7BS5Q) context deadline exceeded ======LOGS====== h24m32.721232367s) [upload=506178]: Done starting upload Dec 29 16:53:32.850331159 (577h24m32.721414194s) [upload=506178]: Uploading segments... Dec 29 16:53:32.850400471 (577h24m32.721483506s) [upload=506178]: Got next segment. Inline: false Dec 29 16:53:32.850436399 (577h24m32.721519474s) [upload=506178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:53:32.850442390 (577h24m32.721525455s) [upload=506178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:53:33.096982539 (577h24m32.968065574s) [seg_pos={0 0}, upload=506178]: Begin upload segment... Dec 29 16:53:33.097039376 (577h24m32.968122411s) [seg_pos={0 0}, upload=506178]: Done begin upload segment. Dec 29 16:53:33.097066108 (577h24m32.968149153s) [noise=true , address=145.79.227.38:4979 , piece_id=ZI3KQ7KYQO6B7YCKNSOWCA5Z4GOEJYRE4ALHDJCXVPFN4JTSV2UQ, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.097248796 (577h24m32.968331831s) [noise=true , address=145.79.227.39:4015 , piece_id=7QWYQN5F2QORPNUDHYGOH2PLUH3MJJ6ILYMTFHOGDTPBQ7EDPD7Q, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.098132288 (577h24m32.969215323s) [noise=true , address=77.90.31.13:4627 , piece_id=2GCBYQ4EQV46G6NQME2QO2HJP62YDXUL3LBAI4MAQQMAAOFFST5A, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.098444573 (577h24m32.969527618s) [noise=true , address=145.79.227.52:4446 , piece_id=4UPQBLTNVP4HWLWWFTJQNJB2MGMEBQKW6CS525W6KN52LGAHHBVQ, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.098562928 (577h24m32.969645963s) [noise=true , address=145.79.227.2:4115 , piece_id=QBTE2C6VUOCLFPEZOZTBW64VH6NFNEWB6JPWNH3V6QOCJEV4GBLQ, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.098731309 (577h24m32.969814354s) [noise=true , address=45.91.116.6:4973 , piece_id=MPZIMNCWWBBVS4CJVDBDVG5VSE2V42674TQG2BSGVC4PCGP7BS5Q, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.098877998 (577h24m32.969961033s) [noise=true , address=45.91.116.4:4937 , piece_id=HMRDH7VEMLACMJ6T7GT5D6GU55EKSXLD6R6EXLOANRH2GIHDQ6TA, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.098984671 (577h24m32.970067716s) [noise=true , address=45.91.116.13:4154 , piece_id=ZAJOOJVDB5OCWJGFKR6R2KHHSSTDCTLFTXGEROAQNJK7YAHOIFIA, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.099153332 (577h24m32.970236367s) [noise=true , address=77.90.31.14:4663 , piece_id=COVIKPHNTGONA7F2DUIZKKZX3CQJJKVMAJAGLYCGIY3LK3MCXDFA, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.099320741 (577h24m32.970403786s) [noise=true , address=145.79.227.26:4653 , piece_id=XACH53DVSNAZCRRKUN6BZEXZW5S6WIF65SDODWAQOSPUNQ5FJH4A, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.099435790 (577h24m32.970518825s) [noise=true , address=145.79.227.37:4566 , piece_id=DXL4VV7LA4PDW57C2SP3JDR5YMO7UOYTHT35O2MSMRV6PSBQ4AQA, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.099640369 (577h24m32.970723405s) [noise=true , address=45.91.116.7:4010 , piece_id=QWTN2EVHLPTYFPDLBSLYVI67BX7QQAPWX6VBWJC3S76HW76M7UCA, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.099796527 (577h24m32.970879562s) [noise=true , address=145.79.227.35:4482 , piece_id=TQD3N747KGFD4BEABBOLVYNIAVSBIX2WJUVNL25FEBHZYSKFU3TA, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.100085668 (577h24m32.971168713s) [noise=true , address=145.79.227.36:4517 , piece_id=AMU6RNBDYI46OPO6BYYRL7PGTPTKA63OD4OQSCS2HF7N255ECOOA, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.100381381 (577h24m32.971464416s) [noise=true , address=45.91.116.11:4119 , piece_id=735KZF5EZLRIWSKKSBIYCPDQLLW45A4QHIIJ35RJPNVR35VYVFIQ, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:33.100678136 (577h24m32.971761171s) [noise=true , address=145.79.227.8:4329 , piece_id=7KXEVJHDO3BGDT7YTQRYQUANMFS4UXTDEC5AIGB2OJ6QMJYHFYKA, seg_pos={0 0}, upload=506178]: Uploading piece... Dec 29 16:53:34.704000359 (577h24m34.575083394s) [upload=506178]: Next returned nil segment Dec 29 16:53:34.704052258 (577h24m34.575135323s) [upload=506178]: Waiting for error group managing segments... Dec 29 16:53:35.087892666 (577h24m34.958975701s) [noise=true , address=145.79.227.39:4015 , piece_id=7QWYQN5F2QORPNUDHYGOH2PLUH3MJJ6ILYMTFHOGDTPBQ7EDPD7Q, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.088487680 (577h24m34.959570715s) [noise=true , address=145.79.227.2:4115 , piece_id=QBTE2C6VUOCLFPEZOZTBW64VH6NFNEWB6JPWNH3V6QOCJEV4GBLQ, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.094842590 (577h24m34.965925635s) [noise=true , address=145.79.227.8:4329 , piece_id=7KXEVJHDO3BGDT7YTQRYQUANMFS4UXTDEC5AIGB2OJ6QMJYHFYKA, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.095719049 (577h24m34.966802094s) [noise=true , address=145.79.227.37:4566 , piece_id=DXL4VV7LA4PDW57C2SP3JDR5YMO7UOYTHT35O2MSMRV6PSBQ4AQA, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.096477063 (577h24m34.967560108s) [noise=true , address=145.79.227.26:4653 , piece_id=XACH53DVSNAZCRRKUN6BZEXZW5S6WIF65SDODWAQOSPUNQ5FJH4A, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.108621748 (577h24m34.979704783s) [noise=true , address=145.79.227.36:4517 , piece_id=AMU6RNBDYI46OPO6BYYRL7PGTPTKA63OD4OQSCS2HF7N255ECOOA, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.109320118 (577h24m34.980403163s) [noise=true , address=145.79.227.52:4446 , piece_id=4UPQBLTNVP4HWLWWFTJQNJB2MGMEBQKW6CS525W6KN52LGAHHBVQ, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.117206054 (577h24m34.988289089s) [noise=true , address=145.79.227.38:4979 , piece_id=ZI3KQ7KYQO6B7YCKNSOWCA5Z4GOEJYRE4ALHDJCXVPFN4JTSV2UQ, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.257907400 (577h24m35.128990435s) [noise=true , address=77.90.31.13:4627 , piece_id=2GCBYQ4EQV46G6NQME2QO2HJP62YDXUL3LBAI4MAQQMAAOFFST5A, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.258327529 (577h24m35.129410565s) [noise=true , address=77.90.31.14:4663 , piece_id=COVIKPHNTGONA7F2DUIZKKZX3CQJJKVMAJAGLYCGIY3LK3MCXDFA, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.340843863 (577h24m35.211926908s) [noise=true , address=45.91.116.7:4010 , piece_id=QWTN2EVHLPTYFPDLBSLYVI67BX7QQAPWX6VBWJC3S76HW76M7UCA, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.353463533 (577h24m35.224546578s) [noise=true , address=45.91.116.13:4154 , piece_id=ZAJOOJVDB5OCWJGFKR6R2KHHSSTDCTLFTXGEROAQNJK7YAHOIFIA, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.364673539 (577h24m35.235756584s) [noise=true , address=45.91.116.6:4973 , piece_id=MPZIMNCWWBBVS4CJVDBDVG5VSE2V42674TQG2BSGVC4PCGP7BS5Q, seg_pos={0 0}, upload=506178]: Done uploading piece. err: <nil> Dec 29 16:53:35.364695560 (577h24m35.235778605s) [seg_pos={0 0}, upload=506178]: Segment reached optimal threshold of 13 pieces. Dec 29 16:53:35.364961928 (577h24m35.236044973s) [noise=true , address=45.91.116.4:4937 , piece_id=HMRDH7VEMLACMJ6T7GT5D6GU55EKSXLD6R6EXLOANRH2GIHDQ6TA, seg_pos={0 0}, upload=506178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:47858->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:47858->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47858->45.91.116.4:4937: use of closed network connection Dec 29 16:53:35.364973239 (577h24m35.236056284s) [noise=true , address=45.91.116.4:4937 , piece_id=HMRDH7VEMLACMJ6T7GT5D6GU55EKSXLD6R6EXLOANRH2GIHDQ6TA, seg_pos={0 0}, upload=506178]: Long tail cancellation due to optimal threshold Dec 29 16:53:35.365106503 (577h24m35.236189548s) [noise=true , address=45.91.116.11:4119 , piece_id=735KZF5EZLRIWSKKSBIYCPDQLLW45A4QHIIJ35RJPNVR35VYVFIQ, seg_pos={0 0}, upload=506178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:44696->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 16:53:35.365117214 (577h24m35.236200259s) [noise=true , address=45.91.116.11:4119 , piece_id=735KZF5EZLRIWSKKSBIYCPDQLLW45A4QHIIJ35RJPNVR35VYVFIQ, seg_pos={0 0}, upload=506178]: Long tail cancellation due to optimal threshold Dec 29 16:53:35.365184641 (577h24m35.236267676s) [noise=true , address=145.79.227.35:4482 , piece_id=TQD3N747KGFD4BEABBOLVYNIAVSBIX2WJUVNL25FEBHZYSKFU3TA, seg_pos={0 0}, upload=506178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 16:53:35.365191455 (577h24m35.23627449s) [noise=true , address=145.79.227.35:4482 , piece_id=TQD3N747KGFD4BEABBOLVYNIAVSBIX2WJUVNL25FEBHZYSKFU3TA, seg_pos={0 0}, upload=506178]: Long tail cancellation due to optimal threshold Dec 29 16:53:35.365254103 (577h24m35.236337149s) [seg_pos={0 0}, upload=506178]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:53:35.365339226 (577h24m35.236422271s) [upload=506178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:53:35.365347672 (577h24m35.236430707s) [upload=506178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:53:35.659150787 (577h24m35.530233832s) [upload=506178]: Done uploading segments... Dec 29 16:53:35.659172809 (577h24m35.530255844s) [upload=506178]: Upload finished. err: <nil> Duration: 30.003366333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.......X..................] duration: 40.589800035s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:53:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 40.589817417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX] duration: 5.001870789s trace: https://jaeger.dev.storj.io/trace/645adc81226b580c FAIL (2025-12-29 16:53:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001898432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.......X.................] duration: 39.8556769s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:52:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 39.855705584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [......X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.] duration: 26.120673597s trace: https://jaeger.dev.storj.io/trace/695cfe3b4e462f15 FAIL (2025-12-29 16:52:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 16:51:46.144871990 (577h22m46.015955025s) [upload=506154]: Got next segment. Inline: false Dec 29 16:51:46.144904191 (577h22m46.015987236s) [upload=506154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:51:46.144940460 (577h22m46.016023505s) [upload=506154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:51:46.392614268 (577h22m46.263697303s) [seg_pos={0 0}, upload=506154]: Begin upload segment... Dec 29 16:51:46.392662139 (577h22m46.263745184s) [seg_pos={0 0}, upload=506154]: Done begin upload segment. Dec 29 16:51:46.392738685 (577h22m46.26382173s) [noise=true , address=77.90.31.18:4808 , piece_id=TB5CVKCAL5JFLCMGYJM647OO6GP5BMOKMES5HTS6R7QAL4N7VDNA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.392763783 (577h22m46.263846988s) [noise=true , address=145.79.227.11:4732 , piece_id=SXMBVZJ4HI2X6BLE2IVFWGM4RIEBJEL3XXLLPXGWEWIRQ3RY2QOA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.393410765 (577h22m46.2644938s) [noise=true , address=145.79.227.51:4409 , piece_id=C4E3HZDDX2ZCLCOFJQ2UMIRPGT2NPIOR46FP3G2B22LA6ZEBXMOQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.393740432 (577h22m46.264823467s) [noise=true , address=45.91.116.5:4298 , piece_id=ROPCE42K6DUODJCPKVSRCOQUUZFRO4VDXOQ3QKSH2QL6L5U4Z6PQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.393786530 (577h22m46.264869565s) [noise=true , address=145.79.227.19:4077 , piece_id=PPQKI6EY6XBLU5YG2FCZFIGUAVYEG6G76DS2XGVTW4RLT2RG4V7Q, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.394039442 (577h22m46.265122477s) [noise=true , address=45.91.116.6:4975 , piece_id=RXYMYBVGIXSMOXAQH3T4N7FB5TPIS3C3UX5NNMGI55J56SP2HDVQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.394300789 (577h22m46.265383824s) [noise=true , address=145.79.227.8:4329 , piece_id=3YUOU3LL3SVQKCUGXL6CJLIXFIFJQZUV4P5FK4UP6FMH3WQF4LBQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.394435606 (577h22m46.265518641s) [noise=true , address=145.79.227.6:4258 , piece_id=CGAF2TDBT6CPIIOQS2D5KUCZDW5IIPUB4Q5CPTOVSGOBKOHX6C4A, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.394885272 (577h22m46.265968307s) [noise=true , address=145.79.227.2:4114 , piece_id=43YWRIV2ZAKMEHESYF6XUEWG2ONMFLC6JBGJQMFR7GZWWGLLYW7A, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.395123135 (577h22m46.266206171s) [noise=true , address=45.91.116.7:4012 , piece_id=3RCGQUZ2VOUKDMP662LBSE7FZYXLTJ35KOB5IDC42H6QV5MC4DVA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.395438376 (577h22m46.266521411s) [noise=true , address=145.79.227.14:4838 , piece_id=6JYJGTMXOCVXBOO6EJ2GPHXG3ILH3MFLJY3UHHO4ELYPK6BOEKMQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.395781028 (577h22m46.266864063s) [noise=true , address=77.90.31.17:4770 , piece_id=MZKAUJTAF6Q5TZQ4GHDVNCTIMQBUABPLUM3KUWG5CUDLZFMCSBLA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.396180078 (577h22m46.267263113s) [noise=true , address=145.79.227.15:4874 , piece_id=4BIWTW2YY6HX3PSQZ3VIHJDXZNYTS5SXLQHPUVDEGD4ETOSFY4BA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.396382483 (577h22m46.267465518s) [noise=true , address=145.79.227.16:4909 , piece_id=4OAUBFAA2INQ4ORYW6PKXMIKBOYJFGFYYSU4HOLWUEG5RCREULHA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.396480741 (577h22m46.267563776s) [noise=true , address=145.79.227.23:4546 , piece_id=ZA7NG5TIT7C7KU22QYTEDHEHD67OBJDXW2SBMGYJYDLIPDA2LRMA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.396746696 (577h22m46.267829731s) [noise=true , address=77.90.31.21:4878 , piece_id=QY53UUQHJDWOTLGMG2OR4T6SYRB6VAUCXVTELMSZKXB4CZDEWRHQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:48.087918458 (577h22m47.959001493s) [upload=506154]: Next returned nil segment Dec 29 16:51:48.087954877 (577h22m47.959037922s) [upload=506154]: Waiting for error group managing segments... Dec 29 16:51:48.369885765 (577h22m48.24096881s) [noise=true , address=145.79.227.14:4838 , piece_id=6JYJGTMXOCVXBOO6EJ2GPHXG3ILH3MFLJY3UHHO4ELYPK6BOEKMQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.374044544 (577h22m48.245127599s) [noise=true , address=145.79.227.6:4258 , piece_id=CGAF2TDBT6CPIIOQS2D5KUCZDW5IIPUB4Q5CPTOVSGOBKOHX6C4A, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.384038884 (577h22m48.255121919s) [noise=true , address=145.79.227.16:4909 , piece_id=4OAUBFAA2INQ4ORYW6PKXMIKBOYJFGFYYSU4HOLWUEG5RCREULHA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.385387523 (577h22m48.256470558s) [noise=true , address=145.79.227.19:4077 , piece_id=PPQKI6EY6XBLU5YG2FCZFIGUAVYEG6G76DS2XGVTW4RLT2RG4V7Q, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.397457406 (577h22m48.268540441s) [noise=true , address=145.79.227.51:4409 , piece_id=C4E3HZDDX2ZCLCOFJQ2UMIRPGT2NPIOR46FP3G2B22LA6ZEBXMOQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.400179810 (577h22m48.271262855s) [noise=true , address=145.79.227.8:4329 , piece_id=3YUOU3LL3SVQKCUGXL6CJLIXFIFJQZUV4P5FK4UP6FMH3WQF4LBQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.402103753 (577h22m48.273186798s) [noise=true , address=145.79.227.23:4546 , piece_id=ZA7NG5TIT7C7KU22QYTEDHEHD67OBJDXW2SBMGYJYDLIPDA2LRMA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.409407941 (577h22m48.280490976s) [noise=true , address=145.79.227.15:4874 , piece_id=4BIWTW2YY6HX3PSQZ3VIHJDXZNYTS5SXLQHPUVDEGD4ETOSFY4BA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.413532826 (577h22m48.284615871s) [noise=true , address=145.79.227.11:4732 , piece_id=SXMBVZJ4HI2X6BLE2IVFWGM4RIEBJEL3XXLLPXGWEWIRQ3RY2QOA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.481269837 (577h22m48.352352872s) [noise=true , address=145.79.227.2:4114 , piece_id=43YWRIV2ZAKMEHESYF6XUEWG2ONMFLC6JBGJQMFR7GZWWGLLYW7A, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.590223821 (577h22m48.461306856s) [noise=true , address=77.90.31.17:4770 , piece_id=MZKAUJTAF6Q5TZQ4GHDVNCTIMQBUABPLUM3KUWG5CUDLZFMCSBLA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.664333096 (577h22m48.535416141s) [noise=true , address=77.90.31.21:4878 , piece_id=QY53UUQHJDWOTLGMG2OR4T6SYRB6VAUCXVTELMSZKXB4CZDEWRHQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.787966092 (577h22m48.659049137s) [noise=true , address=45.91.116.6:4975 , piece_id=RXYMYBVGIXSMOXAQH3T4N7FB5TPIS3C3UX5NNMGI55J56SP2HDVQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.787992733 (577h22m48.659075778s) [seg_pos={0 0}, upload=506154]: Segment reached optimal threshold of 13 pieces. Dec 29 16:51:48.788236958 (577h22m48.659320013s) [noise=true , address=45.91.116.5:4298 , piece_id=ROPCE42K6DUODJCPKVSRCOQUUZFRO4VDXOQ3QKSH2QL6L5U4Z6PQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 16:51:48.788245865 (577h22m48.6593289s) [noise=true , address=45.91.116.5:4298 , piece_id=ROPCE42K6DUODJCPKVSRCOQUUZFRO4VDXOQ3QKSH2QL6L5U4Z6PQ, seg_pos={0 0}, upload=506154]: Long tail cancellation due to optimal threshold Dec 29 16:51:48.788449252 (577h22m48.659532297s) [noise=true , address=77.90.31.18:4808 , piece_id=TB5CVKCAL5JFLCMGYJM647OO6GP5BMOKMES5HTS6R7QAL4N7VDNA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:34916->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:34916->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34916->77.90.31.18:4808: use of closed network connection Dec 29 16:51:48.788469201 (577h22m48.659552236s) [noise=true , address=77.90.31.18:4808 , piece_id=TB5CVKCAL5JFLCMGYJM647OO6GP5BMOKMES5HTS6R7QAL4N7VDNA, seg_pos={0 0}, upload=506154]: Long tail cancellation due to optimal threshold Dec 29 16:51:48.788585031 (577h22m48.659668066s) [noise=true , address=45.91.116.7:4012 , piece_id=3RCGQUZ2VOUKDMP662LBSE7FZYXLTJ35KOB5IDC42H6QV5MC4DVA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:34646->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:34646->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34646->45.91.116.7:4012: use of closed network connection Dec 29 16:51:48.788595240 (577h22m48.659678275s) [noise=true , address=45.91.116.7:4012 , piece_id=3RCGQUZ2VOUKDMP662LBSE7FZYXLTJ35KOB5IDC42H6QV5MC4DVA, seg_pos={0 0}, upload=506154]: Long tail cancellation due to optimal threshold Dec 29 16:51:48.788619557 (577h22m48.659702602s) [seg_pos={0 0}, upload=506154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:51:48.788706762 (577h22m48.659789807s) [upload=506154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:51:48.788713626 (577h22m48.659796671s) [upload=506154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:51:49.083871741 (577h22m48.954954786s) [upload=506154]: Done uploading segments... Dec 29 16:51:49.083895577 (577h22m48.954978612s) [upload=506154]: Upload finished. err: <nil> Duration: 26.120700891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XX] duration: 5.001502598s trace: https://jaeger.dev.storj.io/trace/492a033fd9994b81 FAIL (2025-12-29 16:52:27 UTC) orbiter warning: context deadline exceeded Duration: 5.00152969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X......X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX] duration: 30.003079183s trace: https://jaeger.dev.storj.io/trace/695cfe3b4e462f15 FAIL (2025-12-29 16:52:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 16:51:46.144871990 (577h22m46.015955025s) [upload=506154]: Got next segment. Inline: false Dec 29 16:51:46.144904191 (577h22m46.015987236s) [upload=506154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:51:46.144940460 (577h22m46.016023505s) [upload=506154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:51:46.392614268 (577h22m46.263697303s) [seg_pos={0 0}, upload=506154]: Begin upload segment... Dec 29 16:51:46.392662139 (577h22m46.263745184s) [seg_pos={0 0}, upload=506154]: Done begin upload segment. Dec 29 16:51:46.392738685 (577h22m46.26382173s) [noise=true , address=77.90.31.18:4808 , piece_id=TB5CVKCAL5JFLCMGYJM647OO6GP5BMOKMES5HTS6R7QAL4N7VDNA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.392763783 (577h22m46.263846988s) [noise=true , address=145.79.227.11:4732 , piece_id=SXMBVZJ4HI2X6BLE2IVFWGM4RIEBJEL3XXLLPXGWEWIRQ3RY2QOA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.393410765 (577h22m46.2644938s) [noise=true , address=145.79.227.51:4409 , piece_id=C4E3HZDDX2ZCLCOFJQ2UMIRPGT2NPIOR46FP3G2B22LA6ZEBXMOQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.393740432 (577h22m46.264823467s) [noise=true , address=45.91.116.5:4298 , piece_id=ROPCE42K6DUODJCPKVSRCOQUUZFRO4VDXOQ3QKSH2QL6L5U4Z6PQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.393786530 (577h22m46.264869565s) [noise=true , address=145.79.227.19:4077 , piece_id=PPQKI6EY6XBLU5YG2FCZFIGUAVYEG6G76DS2XGVTW4RLT2RG4V7Q, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.394039442 (577h22m46.265122477s) [noise=true , address=45.91.116.6:4975 , piece_id=RXYMYBVGIXSMOXAQH3T4N7FB5TPIS3C3UX5NNMGI55J56SP2HDVQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.394300789 (577h22m46.265383824s) [noise=true , address=145.79.227.8:4329 , piece_id=3YUOU3LL3SVQKCUGXL6CJLIXFIFJQZUV4P5FK4UP6FMH3WQF4LBQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.394435606 (577h22m46.265518641s) [noise=true , address=145.79.227.6:4258 , piece_id=CGAF2TDBT6CPIIOQS2D5KUCZDW5IIPUB4Q5CPTOVSGOBKOHX6C4A, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.394885272 (577h22m46.265968307s) [noise=true , address=145.79.227.2:4114 , piece_id=43YWRIV2ZAKMEHESYF6XUEWG2ONMFLC6JBGJQMFR7GZWWGLLYW7A, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.395123135 (577h22m46.266206171s) [noise=true , address=45.91.116.7:4012 , piece_id=3RCGQUZ2VOUKDMP662LBSE7FZYXLTJ35KOB5IDC42H6QV5MC4DVA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.395438376 (577h22m46.266521411s) [noise=true , address=145.79.227.14:4838 , piece_id=6JYJGTMXOCVXBOO6EJ2GPHXG3ILH3MFLJY3UHHO4ELYPK6BOEKMQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.395781028 (577h22m46.266864063s) [noise=true , address=77.90.31.17:4770 , piece_id=MZKAUJTAF6Q5TZQ4GHDVNCTIMQBUABPLUM3KUWG5CUDLZFMCSBLA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.396180078 (577h22m46.267263113s) [noise=true , address=145.79.227.15:4874 , piece_id=4BIWTW2YY6HX3PSQZ3VIHJDXZNYTS5SXLQHPUVDEGD4ETOSFY4BA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.396382483 (577h22m46.267465518s) [noise=true , address=145.79.227.16:4909 , piece_id=4OAUBFAA2INQ4ORYW6PKXMIKBOYJFGFYYSU4HOLWUEG5RCREULHA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.396480741 (577h22m46.267563776s) [noise=true , address=145.79.227.23:4546 , piece_id=ZA7NG5TIT7C7KU22QYTEDHEHD67OBJDXW2SBMGYJYDLIPDA2LRMA, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:46.396746696 (577h22m46.267829731s) [noise=true , address=77.90.31.21:4878 , piece_id=QY53UUQHJDWOTLGMG2OR4T6SYRB6VAUCXVTELMSZKXB4CZDEWRHQ, seg_pos={0 0}, upload=506154]: Uploading piece... Dec 29 16:51:48.087918458 (577h22m47.959001493s) [upload=506154]: Next returned nil segment Dec 29 16:51:48.087954877 (577h22m47.959037922s) [upload=506154]: Waiting for error group managing segments... Dec 29 16:51:48.369885765 (577h22m48.24096881s) [noise=true , address=145.79.227.14:4838 , piece_id=6JYJGTMXOCVXBOO6EJ2GPHXG3ILH3MFLJY3UHHO4ELYPK6BOEKMQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.374044544 (577h22m48.245127599s) [noise=true , address=145.79.227.6:4258 , piece_id=CGAF2TDBT6CPIIOQS2D5KUCZDW5IIPUB4Q5CPTOVSGOBKOHX6C4A, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.384038884 (577h22m48.255121919s) [noise=true , address=145.79.227.16:4909 , piece_id=4OAUBFAA2INQ4ORYW6PKXMIKBOYJFGFYYSU4HOLWUEG5RCREULHA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.385387523 (577h22m48.256470558s) [noise=true , address=145.79.227.19:4077 , piece_id=PPQKI6EY6XBLU5YG2FCZFIGUAVYEG6G76DS2XGVTW4RLT2RG4V7Q, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.397457406 (577h22m48.268540441s) [noise=true , address=145.79.227.51:4409 , piece_id=C4E3HZDDX2ZCLCOFJQ2UMIRPGT2NPIOR46FP3G2B22LA6ZEBXMOQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.400179810 (577h22m48.271262855s) [noise=true , address=145.79.227.8:4329 , piece_id=3YUOU3LL3SVQKCUGXL6CJLIXFIFJQZUV4P5FK4UP6FMH3WQF4LBQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.402103753 (577h22m48.273186798s) [noise=true , address=145.79.227.23:4546 , piece_id=ZA7NG5TIT7C7KU22QYTEDHEHD67OBJDXW2SBMGYJYDLIPDA2LRMA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.409407941 (577h22m48.280490976s) [noise=true , address=145.79.227.15:4874 , piece_id=4BIWTW2YY6HX3PSQZ3VIHJDXZNYTS5SXLQHPUVDEGD4ETOSFY4BA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.413532826 (577h22m48.284615871s) [noise=true , address=145.79.227.11:4732 , piece_id=SXMBVZJ4HI2X6BLE2IVFWGM4RIEBJEL3XXLLPXGWEWIRQ3RY2QOA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.481269837 (577h22m48.352352872s) [noise=true , address=145.79.227.2:4114 , piece_id=43YWRIV2ZAKMEHESYF6XUEWG2ONMFLC6JBGJQMFR7GZWWGLLYW7A, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.590223821 (577h22m48.461306856s) [noise=true , address=77.90.31.17:4770 , piece_id=MZKAUJTAF6Q5TZQ4GHDVNCTIMQBUABPLUM3KUWG5CUDLZFMCSBLA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.664333096 (577h22m48.535416141s) [noise=true , address=77.90.31.21:4878 , piece_id=QY53UUQHJDWOTLGMG2OR4T6SYRB6VAUCXVTELMSZKXB4CZDEWRHQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.787966092 (577h22m48.659049137s) [noise=true , address=45.91.116.6:4975 , piece_id=RXYMYBVGIXSMOXAQH3T4N7FB5TPIS3C3UX5NNMGI55J56SP2HDVQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: <nil> Dec 29 16:51:48.787992733 (577h22m48.659075778s) [seg_pos={0 0}, upload=506154]: Segment reached optimal threshold of 13 pieces. Dec 29 16:51:48.788236958 (577h22m48.659320013s) [noise=true , address=45.91.116.5:4298 , piece_id=ROPCE42K6DUODJCPKVSRCOQUUZFRO4VDXOQ3QKSH2QL6L5U4Z6PQ, seg_pos={0 0}, upload=506154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 16:51:48.788245865 (577h22m48.6593289s) [noise=true , address=45.91.116.5:4298 , piece_id=ROPCE42K6DUODJCPKVSRCOQUUZFRO4VDXOQ3QKSH2QL6L5U4Z6PQ, seg_pos={0 0}, upload=506154]: Long tail cancellation due to optimal threshold Dec 29 16:51:48.788449252 (577h22m48.659532297s) [noise=true , address=77.90.31.18:4808 , piece_id=TB5CVKCAL5JFLCMGYJM647OO6GP5BMOKMES5HTS6R7QAL4N7VDNA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:34916->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:34916->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34916->77.90.31.18:4808: use of closed network connection Dec 29 16:51:48.788469201 (577h22m48.659552236s) [noise=true , address=77.90.31.18:4808 , piece_id=TB5CVKCAL5JFLCMGYJM647OO6GP5BMOKMES5HTS6R7QAL4N7VDNA, seg_pos={0 0}, upload=506154]: Long tail cancellation due to optimal threshold Dec 29 16:51:48.788585031 (577h22m48.659668066s) [noise=true , address=45.91.116.7:4012 , piece_id=3RCGQUZ2VOUKDMP662LBSE7FZYXLTJ35KOB5IDC42H6QV5MC4DVA, seg_pos={0 0}, upload=506154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:34646->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:34646->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34646->45.91.116.7:4012: use of closed network connection Dec 29 16:51:48.788595240 (577h22m48.659678275s) [noise=true , address=45.91.116.7:4012 , piece_id=3RCGQUZ2VOUKDMP662LBSE7FZYXLTJ35KOB5IDC42H6QV5MC4DVA, seg_pos={0 0}, upload=506154]: Long tail cancellation due to optimal threshold Dec 29 16:51:48.788619557 (577h22m48.659702602s) [seg_pos={0 0}, upload=506154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:51:48.788706762 (577h22m48.659789807s) [upload=506154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:51:48.788713626 (577h22m48.659796671s) [upload=506154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:51:49.083871741 (577h22m48.954954786s) [upload=506154]: Done uploading segments... Dec 29 16:51:49.083895577 (577h22m48.954978612s) [upload=506154]: Upload finished. err: <nil> Duration: 30.00309391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.......X................] duration: 33.575659458s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:52:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 33.575677361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.......X...............] duration: 43.19226017s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:51:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 43.19228593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.X] duration: 5.002141698s trace: https://jaeger.dev.storj.io/trace/11fc64cf7541a41b FAIL (2025-12-29 16:51:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002167717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X......X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXX] duration: 30.003984889s trace: https://jaeger.dev.storj.io/trace/cb4cb91998bd0df FAIL (2025-12-29 16:51:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: HANO5AYBO3X6PMJ6YMZ2Z2O6QXRRDKTOASJQ7L3R64W6RJIBGE4Q) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: AAQLMYRWX2CDTCPBIPCB46BUG3CVRNVFWZSTMVWS3BDY7PBCHMEQ) context deadline exceeded ======LOGS====== 29 16:50:45.260281058 (577h21m45.131364093s) [upload=506138]: Starting upload Dec 29 16:50:45.260344348 (577h21m45.131427383s) [upload=506138]: Done starting upload Dec 29 16:50:45.260579597 (577h21m45.131662652s) [upload=506138]: Uploading segments... Dec 29 16:50:45.260714824 (577h21m45.131797859s) [upload=506138]: Got next segment. Inline: false Dec 29 16:50:45.260763286 (577h21m45.131846321s) [upload=506138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:50:45.260770129 (577h21m45.131853164s) [upload=506138]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:50:45.511223560 (577h21m45.382306605s) [seg_pos={0 0}, upload=506138]: Begin upload segment... Dec 29 16:50:45.511371662 (577h21m45.382454697s) [seg_pos={0 0}, upload=506138]: Done begin upload segment. Dec 29 16:50:45.511420674 (577h21m45.38250371s) [noise=true , address=45.91.116.8:4046 , piece_id=2CQOOILOZEVDIR4CR6VN6PBWJO2LQJN4PCWXNTCCKGF6JCAHSXQA, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.511461042 (577h21m45.382544087s) [noise=true , address=145.79.227.51:4409 , piece_id=QXUGFFTLKJ2ZJMUPSNJEIXN574QONZ7RQI5664Z2A4SKQ65MYA5Q, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.512042679 (577h21m45.383125724s) [noise=true , address=145.79.227.50:4375 , piece_id=N6YZIUZCVZQGGC5LKGHHOOYTXV6RITAW5F4G3GYJ77AWD2FFJVBA, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.512116920 (577h21m45.383199965s) [noise=true , address=145.79.227.42:4123 , piece_id=G62JPGKFSRAREH7Q7SFFKEWE6Y4QXDKZ6N66CLYXSXOUE7I5AMHA, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.512476395 (577h21m45.38355943s) [noise=true , address=77.90.31.11:4555 , piece_id=JSDM2YZ3HIFRZTIHEMRHXYYEIQZOU64HKCUOLR73GCFSIWYGCVIQ, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.512659794 (577h21m45.383742829s) [noise=true , address=145.79.227.26:4655 , piece_id=WDCUBKAYH7V5QN2E5DRP4425VBS4QPWK5PUY6FYVQBVDPLL25FNA, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.512860677 (577h21m45.383943722s) [noise=true , address=45.91.116.7:4009 , piece_id=AAQLMYRWX2CDTCPBIPCB46BUG3CVRNVFWZSTMVWS3BDY7PBCHMEQ, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.513331874 (577h21m45.384414909s) [noise=true , address=145.79.227.43:4159 , piece_id=YO4QAP5UHF4VPNBWNLVEUDEXPVH5VUISD7H2N3C657BSMEJIT6DQ, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.513385336 (577h21m45.384468371s) [noise=true , address=145.79.227.39:4015 , piece_id=EPUTSUT23HASTOQSIKSZDBTCC6ST3JPGFYIQRZ3DUTSCQHDA4T6Q, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.513705626 (577h21m45.384788661s) [noise=true , address=145.79.227.14:4839 , piece_id=QU75LE5CXRAB6LVRWZOSE6TGHK5WJN3KFAQZGXSI4WWP3XHX2XUA, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.513760690 (577h21m45.384843735s) [noise=true , address=45.91.116.13:4155 , piece_id=I7MCQN72KKKYF7PMM5AR4RGB4QVLEUYCDZFBXVTZP6Y4LF7WTTNA, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.514288897 (577h21m45.385371932s) [noise=true , address=145.79.227.24:4582 , piece_id=QKOLKMRBTT67DSEFY34L53UJ4CYTNNE2Z3NKBCJODO5SLTCSOZZA, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.514880673 (577h21m45.385963709s) [noise=true , address=45.91.116.11:4117 , piece_id=HANO5AYBO3X6PMJ6YMZ2Z2O6QXRRDKTOASJQ7L3R64W6RJIBGE4Q, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.515486256 (577h21m45.386569291s) [noise=true , address=145.79.227.23:4546 , piece_id=RR6XUEYQH7TU6S7CAHDKYD7T4EXENW7G4L7XOPTUCDUDMDWN6FXA, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.516032026 (577h21m45.387115061s) [noise=true , address=145.79.227.16:4910 , piece_id=JVQUD5NWSY7ZV6SPJIW6KMOF5TIP32B32O65QLD2PBXTACDKXKJQ, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:45.516671142 (577h21m45.387754177s) [noise=true , address=145.79.227.4:4185 , piece_id=N4LACMKWGYF2263V5VKAV3IADSGCMQZQDYPD36QSGXRCM5R2QW6Q, seg_pos={0 0}, upload=506138]: Uploading piece... Dec 29 16:50:47.203582931 (577h21m47.074665986s) [upload=506138]: Next returned nil segment Dec 29 16:50:47.203601317 (577h21m47.074684372s) [upload=506138]: Waiting for error group managing segments... Dec 29 16:50:47.483331154 (577h21m47.354414189s) [noise=true , address=145.79.227.26:4655 , piece_id=WDCUBKAYH7V5QN2E5DRP4425VBS4QPWK5PUY6FYVQBVDPLL25FNA, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.492462241 (577h21m47.363545276s) [noise=true , address=145.79.227.50:4375 , piece_id=N6YZIUZCVZQGGC5LKGHHOOYTXV6RITAW5F4G3GYJ77AWD2FFJVBA, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.497135990 (577h21m47.368219025s) [noise=true , address=145.79.227.42:4123 , piece_id=G62JPGKFSRAREH7Q7SFFKEWE6Y4QXDKZ6N66CLYXSXOUE7I5AMHA, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.499138023 (577h21m47.370221078s) [noise=true , address=145.79.227.51:4409 , piece_id=QXUGFFTLKJ2ZJMUPSNJEIXN574QONZ7RQI5664Z2A4SKQ65MYA5Q, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.500525825 (577h21m47.37160889s) [noise=true , address=145.79.227.4:4185 , piece_id=N4LACMKWGYF2263V5VKAV3IADSGCMQZQDYPD36QSGXRCM5R2QW6Q, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.501186953 (577h21m47.372269989s) [noise=true , address=145.79.227.14:4839 , piece_id=QU75LE5CXRAB6LVRWZOSE6TGHK5WJN3KFAQZGXSI4WWP3XHX2XUA, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.517237557 (577h21m47.388320602s) [noise=true , address=145.79.227.39:4015 , piece_id=EPUTSUT23HASTOQSIKSZDBTCC6ST3JPGFYIQRZ3DUTSCQHDA4T6Q, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.517419102 (577h21m47.388502147s) [noise=true , address=145.79.227.24:4582 , piece_id=QKOLKMRBTT67DSEFY34L53UJ4CYTNNE2Z3NKBCJODO5SLTCSOZZA, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.528757031 (577h21m47.399840066s) [noise=true , address=145.79.227.23:4546 , piece_id=RR6XUEYQH7TU6S7CAHDKYD7T4EXENW7G4L7XOPTUCDUDMDWN6FXA, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.547395241 (577h21m47.418478276s) [noise=true , address=145.79.227.16:4910 , piece_id=JVQUD5NWSY7ZV6SPJIW6KMOF5TIP32B32O65QLD2PBXTACDKXKJQ, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.845509084 (577h21m47.716592119s) [noise=true , address=45.91.116.13:4155 , piece_id=I7MCQN72KKKYF7PMM5AR4RGB4QVLEUYCDZFBXVTZP6Y4LF7WTTNA, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.868072859 (577h21m47.739155894s) [noise=true , address=45.91.116.7:4009 , piece_id=AAQLMYRWX2CDTCPBIPCB46BUG3CVRNVFWZSTMVWS3BDY7PBCHMEQ, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.900792302 (577h21m47.771875347s) [noise=true , address=45.91.116.11:4117 , piece_id=HANO5AYBO3X6PMJ6YMZ2Z2O6QXRRDKTOASJQ7L3R64W6RJIBGE4Q, seg_pos={0 0}, upload=506138]: Done uploading piece. err: <nil> Dec 29 16:50:47.900826156 (577h21m47.771909191s) [seg_pos={0 0}, upload=506138]: Segment reached optimal threshold of 13 pieces. Dec 29 16:50:47.901122540 (577h21m47.772205625s) [noise=true , address=145.79.227.43:4159 , piece_id=YO4QAP5UHF4VPNBWNLVEUDEXPVH5VUISD7H2N3C657BSMEJIT6DQ, seg_pos={0 0}, upload=506138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 16:50:47.901133321 (577h21m47.772216366s) [noise=true , address=145.79.227.43:4159 , piece_id=YO4QAP5UHF4VPNBWNLVEUDEXPVH5VUISD7H2N3C657BSMEJIT6DQ, seg_pos={0 0}, upload=506138]: Long tail cancellation due to optimal threshold Dec 29 16:50:47.901303504 (577h21m47.77238655s) [noise=true , address=45.91.116.8:4046 , piece_id=2CQOOILOZEVDIR4CR6VN6PBWJO2LQJN4PCWXNTCCKGF6JCAHSXQA, seg_pos={0 0}, upload=506138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 16:50:47.901316800 (577h21m47.772399845s) [noise=true , address=45.91.116.8:4046 , piece_id=2CQOOILOZEVDIR4CR6VN6PBWJO2LQJN4PCWXNTCCKGF6JCAHSXQA, seg_pos={0 0}, upload=506138]: Long tail cancellation due to optimal threshold Dec 29 16:50:47.901324154 (577h21m47.772407219s) [noise=true , address=77.90.31.11:4555 , piece_id=JSDM2YZ3HIFRZTIHEMRHXYYEIQZOU64HKCUOLR73GCFSIWYGCVIQ, seg_pos={0 0}, upload=506138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:60554->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:60554->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60554->77.90.31.11:4555: use of closed network connection Dec 29 16:50:47.901331508 (577h21m47.772414543s) [noise=true , address=77.90.31.11:4555 , piece_id=JSDM2YZ3HIFRZTIHEMRHXYYEIQZOU64HKCUOLR73GCFSIWYGCVIQ, seg_pos={0 0}, upload=506138]: Long tail cancellation due to optimal threshold Dec 29 16:50:47.901349082 (577h21m47.772432117s) [seg_pos={0 0}, upload=506138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:50:47.901431218 (577h21m47.772514273s) [upload=506138]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:50:47.901437800 (577h21m47.772520835s) [upload=506138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:50:48.207211939 (577h21m48.078294984s) [upload=506138]: Done uploading segments... Dec 29 16:50:48.207234442 (577h21m48.078317477s) [upload=506138]: Upload finished. err: <nil> Duration: 30.004011519s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.] duration: 4.31794716s trace: https://jaeger.dev.storj.io/trace/28ad844f4e79c28e FAIL (2025-12-29 16:50:56 UTC) orbiter warning: context deadline exceeded Duration: 4.317977838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.......X..............] duration: 24.707597792s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:50:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 24.707655652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..X......X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XX] duration: 30.001847464s trace: https://jaeger.dev.storj.io/trace/734a07e912dc880a FAIL (2025-12-29 16:50:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 662ZJOUTDXG2CTMLLK3GVO4NRVJKBC5WJD3EDKZDXRX6G7KLSN7Q) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BXUQLXMCCLHOKDMJMJ73J7BKOEGPKCLPTNDX4W6SQJYT2MOU4L6A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6WNSDATJHF7R4QCKMLYDHJSQBQ3RHXXCB2YCLMP5UYTNWZUSRHHA) context deadline exceeded ======LOGS====== Dec 29 16:50:09.671912623 (577h21m9.542995659s) [upload=506127]: Starting upload Dec 29 16:50:09.672011081 (577h21m9.543094116s) [upload=506127]: Done starting upload Dec 29 16:50:09.672086935 (577h21m9.54316998s) [upload=506127]: Uploading segments... Dec 29 16:50:09.672290303 (577h21m9.543373338s) [upload=506127]: Got next segment. Inline: false Dec 29 16:50:09.672398899 (577h21m9.543481935s) [upload=506127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:50:09.672408829 (577h21m9.543491874s) [upload=506127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:50:09.919522019 (577h21m9.790605054s) [seg_pos={0 0}, upload=506127]: Begin upload segment... Dec 29 16:50:09.919575110 (577h21m9.790658155s) [seg_pos={0 0}, upload=506127]: Done begin upload segment. Dec 29 16:50:09.919605117 (577h21m9.790688152s) [noise=true , address=145.79.227.26:4654 , piece_id=CY5PEZTBUY6ESKKRDBUCO7354KUZBKECEHU6FWCAXNK4DRIYVSZQ, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.920103055 (577h21m9.7911861s) [noise=true , address=145.79.227.3:4149 , piece_id=QD6EJ4QV6XVECO7GPVOIWYSR7FLOPRBDCIINFCL3SRJOYPHF5XSQ, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.921014891 (577h21m9.792097936s) [noise=true , address=77.90.31.21:4877 , piece_id=LBWUB2EI6C3UHITWGUPZQ5E4SF2DCCH3GSXGZSYUCH55VOCHLY4A, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.921499374 (577h21m9.792582409s) [noise=true , address=45.91.116.13:4154 , piece_id=BXUQLXMCCLHOKDMJMJ73J7BKOEGPKCLPTNDX4W6SQJYT2MOU4L6A, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.922001850 (577h21m9.793084885s) [noise=true , address=45.91.116.10:4081 , piece_id=BOZ4MN6KTJIYFM47RWDN6CWBF7Z6YTFQA4OB4INFABJ45JJCQQ7A, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.922090119 (577h21m9.793173164s) [noise=true , address=77.90.31.10:4517 , piece_id=6WNSDATJHF7R4QCKMLYDHJSQBQ3RHXXCB2YCLMP5UYTNWZUSRHHA, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.922691173 (577h21m9.793774208s) [noise=true , address=145.79.227.7:4293 , piece_id=2W265Z76RCIZ322XWW6J5TZRDHYBCRZ3EBS7XRBU2RYMAYOU6MHQ, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.922849836 (577h21m9.793932881s) [noise=true , address=145.79.227.12:4765 , piece_id=RBOWRKLORNSVYVMWMCTP53V3K6IZVZVLZ2YDYIKEW5WEYQNSGGSA, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.923158644 (577h21m9.794241679s) [noise=true , address=77.90.31.18:4808 , piece_id=AJIGLDA3TTZ25OSPJGHZ4RBHKANPNW54YRZUK2Q7V5PVUMDC5BOQ, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.923611125 (577h21m9.79469417s) [noise=true , address=45.91.116.7:4011 , piece_id=L5B5AYS7JRLTAN52CV5PVMVNFIVH3RYNKKNAQ34BOBTWEKRAQLNA, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.923644900 (577h21m9.794727935s) [noise=true , address=145.79.227.14:4838 , piece_id=5VK2XRCBAKAAN4OZ43DE5VXRBRDZVKJIKD5IEGGRAZZZ4TDV7A6Q, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.923979166 (577h21m9.795062211s) [noise=true , address=145.79.227.22:4509 , piece_id=46E3HZ6K6GVFUIACMUMJR6HBUEXTRAHKV4KXD7PP64YX3XG2NUIA, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.924292482 (577h21m9.795375527s) [noise=true , address=45.91.116.11:4118 , piece_id=662ZJOUTDXG2CTMLLK3GVO4NRVJKBC5WJD3EDKZDXRX6G7KLSN7Q, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.924343089 (577h21m9.795426134s) [noise=true , address=145.79.227.46:4601 , piece_id=7HIH6HDRPIQSDRRXI3STJEZVQSA5EUI3OPRFGNGUVWTYPPJABWSQ, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.924718553 (577h21m9.795801588s) [noise=true , address=145.79.227.32:4836 , piece_id=XUP7JEJMCOSWUETGBEDJOBLGBBNRERY3ED5XEQSFYDHM5FVNR4TQ, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:09.924995501 (577h21m9.796078546s) [noise=true , address=145.79.227.30:4761 , piece_id=Y3ZEMH6LRJQ7JDYWEJGO5OHPIWPHKRLYNWXRGWR4ZPSV2UTAQJYA, seg_pos={0 0}, upload=506127]: Uploading piece... Dec 29 16:50:11.618609041 (577h21m11.489692086s) [upload=506127]: Next returned nil segment Dec 29 16:50:11.618624992 (577h21m11.489708037s) [upload=506127]: Waiting for error group managing segments... Dec 29 16:50:11.907331612 (577h21m11.778414647s) [noise=true , address=145.79.227.26:4654 , piece_id=CY5PEZTBUY6ESKKRDBUCO7354KUZBKECEHU6FWCAXNK4DRIYVSZQ, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:11.915604084 (577h21m11.78668713s) [noise=true , address=145.79.227.22:4509 , piece_id=46E3HZ6K6GVFUIACMUMJR6HBUEXTRAHKV4KXD7PP64YX3XG2NUIA, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:11.925302682 (577h21m11.796385717s) [noise=true , address=145.79.227.46:4601 , piece_id=7HIH6HDRPIQSDRRXI3STJEZVQSA5EUI3OPRFGNGUVWTYPPJABWSQ, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:11.927907271 (577h21m11.798990317s) [noise=true , address=145.79.227.7:4293 , piece_id=2W265Z76RCIZ322XWW6J5TZRDHYBCRZ3EBS7XRBU2RYMAYOU6MHQ, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:11.928717885 (577h21m11.79980093s) [noise=true , address=145.79.227.12:4765 , piece_id=RBOWRKLORNSVYVMWMCTP53V3K6IZVZVLZ2YDYIKEW5WEYQNSGGSA, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:11.930868851 (577h21m11.801951896s) [noise=true , address=145.79.227.32:4836 , piece_id=XUP7JEJMCOSWUETGBEDJOBLGBBNRERY3ED5XEQSFYDHM5FVNR4TQ, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:11.948685185 (577h21m11.81976823s) [noise=true , address=145.79.227.14:4838 , piece_id=5VK2XRCBAKAAN4OZ43DE5VXRBRDZVKJIKD5IEGGRAZZZ4TDV7A6Q, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:12.107927556 (577h21m11.979010591s) [noise=true , address=77.90.31.21:4877 , piece_id=LBWUB2EI6C3UHITWGUPZQ5E4SF2DCCH3GSXGZSYUCH55VOCHLY4A, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:12.176428391 (577h21m12.047511426s) [noise=true , address=77.90.31.10:4517 , piece_id=6WNSDATJHF7R4QCKMLYDHJSQBQ3RHXXCB2YCLMP5UYTNWZUSRHHA, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:12.188343279 (577h21m12.059426324s) [noise=true , address=45.91.116.11:4118 , piece_id=662ZJOUTDXG2CTMLLK3GVO4NRVJKBC5WJD3EDKZDXRX6G7KLSN7Q, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:12.189075503 (577h21m12.060158538s) [noise=true , address=45.91.116.7:4011 , piece_id=L5B5AYS7JRLTAN52CV5PVMVNFIVH3RYNKKNAQ34BOBTWEKRAQLNA, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:12.234184117 (577h21m12.105267152s) [noise=true , address=145.79.227.3:4149 , piece_id=QD6EJ4QV6XVECO7GPVOIWYSR7FLOPRBDCIINFCL3SRJOYPHF5XSQ, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:12.301327488 (577h21m12.172410533s) [noise=true , address=45.91.116.13:4154 , piece_id=BXUQLXMCCLHOKDMJMJ73J7BKOEGPKCLPTNDX4W6SQJYT2MOU4L6A, seg_pos={0 0}, upload=506127]: Done uploading piece. err: <nil> Dec 29 16:50:12.301353588 (577h21m12.172436633s) [seg_pos={0 0}, upload=506127]: Segment reached optimal threshold of 13 pieces. Dec 29 16:50:12.301594497 (577h21m12.172677542s) [noise=true , address=45.91.116.10:4081 , piece_id=BOZ4MN6KTJIYFM47RWDN6CWBF7Z6YTFQA4OB4INFABJ45JJCQQ7A, seg_pos={0 0}, upload=506127]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 29 16:50:12.301603403 (577h21m12.172686448s) [noise=true , address=45.91.116.10:4081 , piece_id=BOZ4MN6KTJIYFM47RWDN6CWBF7Z6YTFQA4OB4INFABJ45JJCQQ7A, seg_pos={0 0}, upload=506127]: Long tail cancellation due to optimal threshold Dec 29 16:50:12.301774309 (577h21m12.172857354s) [noise=true , address=145.79.227.30:4761 , piece_id=Y3ZEMH6LRJQ7JDYWEJGO5OHPIWPHKRLYNWXRGWR4ZPSV2UTAQJYA, seg_pos={0 0}, upload=506127]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 16:50:12.301785550 (577h21m12.172868605s) [noise=true , address=145.79.227.30:4761 , piece_id=Y3ZEMH6LRJQ7JDYWEJGO5OHPIWPHKRLYNWXRGWR4ZPSV2UTAQJYA, seg_pos={0 0}, upload=506127]: Long tail cancellation due to optimal threshold Dec 29 16:50:12.302020347 (577h21m12.173103402s) [noise=true , address=77.90.31.18:4808 , piece_id=AJIGLDA3TTZ25OSPJGHZ4RBHKANPNW54YRZUK2Q7V5PVUMDC5BOQ, seg_pos={0 0}, upload=506127]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: read tcp 188.166.225.170:40980->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 16:50:12.302030446 (577h21m12.173113481s) [noise=true , address=77.90.31.18:4808 , piece_id=AJIGLDA3TTZ25OSPJGHZ4RBHKANPNW54YRZUK2Q7V5PVUMDC5BOQ, seg_pos={0 0}, upload=506127]: Long tail cancellation due to optimal threshold Dec 29 16:50:12.302052078 (577h21m12.173135113s) [seg_pos={0 0}, upload=506127]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:50:12.302142430 (577h21m12.173225485s) [upload=506127]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:50:12.302148401 (577h21m12.173231447s) [upload=506127]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:50:12.606320740 (577h21m12.477403795s) [upload=506127]: Done uploading segments... Dec 29 16:50:12.606344976 (577h21m12.477428021s) [upload=506127]: Upload finished. err: <nil> Duration: 30.001864747s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X] duration: 5.001772765s trace: https://jaeger.dev.storj.io/trace/28ad844f4e79c28e FAIL (2025-12-29 16:50:27 UTC) orbiter warning: context deadline exceeded Duration: 5.001796351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.......X.............] duration: 27.88457482s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 27.88459557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.......X............] duration: 54.933325185s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:49:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 54.933362987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [X.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.] duration: 3.338985289s trace: https://jaeger.dev.storj.io/trace/1469525d8efa4352 FAIL (2025-12-29 16:49:28 UTC) orbiter warning: context deadline exceeded Duration: 3.339015717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [...X......X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.X] duration: 30.003353686s trace: https://jaeger.dev.storj.io/trace/562283be38cbc9b3 FAIL (2025-12-29 16:49:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 069775007 (577h19m54.940858052s) [upload=506107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:48:55.319605831 (577h19m55.190688866s) [seg_pos={0 0}, upload=506107]: Begin upload segment... Dec 29 16:48:55.319647992 (577h19m55.190731028s) [seg_pos={0 0}, upload=506107]: Done begin upload segment. Dec 29 16:48:55.319674422 (577h19m55.190757457s) [noise=true , address=77.90.31.14:4663 , piece_id=W3XKR53ETYTXKZEEJEX4RYLYTMGJY473576QWJA3JOBCVYHQUTSA, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.319704399 (577h19m55.190787444s) [noise=true , address=45.91.116.7:4012 , piece_id=W4BUUKSBJY4YQ277W3JATBTJ3A6BQZWFOTMLQRSCOR7G5NKG6ZJA, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.320148355 (577h19m55.1912314s) [noise=true , address=145.79.227.51:4411 , piece_id=JIQSH4B4OE6EJAIAMWEZ2HESJS7RZLTR7H63F7FQZPINRESQF7RQ, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.320521215 (577h19m55.19160425s) [noise=true , address=145.79.227.14:4839 , piece_id=TPKXYUKJSGOJ3J5YRAZI47NHBTDDAWQKIRR74LALWXZC2BUDYCCA, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.321073386 (577h19m55.192156432s) [noise=true , address=45.91.116.12:4228 , piece_id=NNWWHABCQNLVV5JMCENA5O4Y7KLZKTUP7T6QO7H3BJWUXSOBAD7A, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.321460654 (577h19m55.192543689s) [noise=true , address=145.79.227.39:4015 , piece_id=R2AXME6L4F5JERMAXDQZGYSK7EPPPP2KIMMSILT55FOH4TMAYZSQ, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.321937852 (577h19m55.193020897s) [noise=true , address=145.79.227.26:4655 , piece_id=ZF6O7GXLW5PMUAYRA4GAVQI3WY7TGTT6SUHVZ4CPN3MREGTB54LQ, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.322335860 (577h19m55.193418895s) [noise=true , address=145.79.227.3:4149 , piece_id=UZW6P4WWORFPOMJ6RVEVEZY73DI7XN5OJPOL3R4CYYARX426LR3Q, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.323360100 (577h19m55.194443135s) [noise=true , address=145.79.227.41:4086 , piece_id=HQGAPMW36KO6V6NMMJDS2GA63VG4HHBL5FIDXUEWK4IP42Z7QMIQ, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.323814816 (577h19m55.194897861s) [noise=true , address=145.79.227.17:4972 , piece_id=DBZRK653HBWZSUA4FUZFCSPOROW7GCR4SZWOV4TRFDKSMI6PTZDQ, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.324529528 (577h19m55.195612563s) [noise=true , address=45.91.116.11:4120 , piece_id=WJ4ZV4I3ITMKLEBFJY4MREKUIWX5PP5O2HJ5ZAXMH5VQ6ZGTEXMA, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.325778937 (577h19m55.196861972s) [noise=true , address=145.79.227.38:4978 , piece_id=7DXGYJRU7EGSRQ2AKBH4MMSBWZGIY7BCTDKXUEANE4Z6FUBN2MXQ, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.327528157 (577h19m55.198611192s) [noise=true , address=45.91.116.14:4191 , piece_id=5622LCYIVFSFQNUBD5N7DZPRQO3KHA7YUYHJ27QI2XWGCOQ7IVEQ, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.328920909 (577h19m55.200003944s) [noise=true , address=145.79.227.50:4374 , piece_id=7CWHTWJ67GSASRNCMNHXTHDZDSSMNJORRRCGWB2JXCMSWQPMSX5Q, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.330308621 (577h19m55.201391656s) [noise=true , address=145.79.227.48:4302 , piece_id=CFNGGADOG67ALTSWARZW75KTBLJYUMTGHO7C3WN3SRS4XTNOI2JA, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:55.330751835 (577h19m55.20183488s) [noise=true , address=145.79.227.7:4295 , piece_id=F5BCBL7DIYDQHNZIDWOHYSVFCTD2XLLO3JQXBRGBZZRIMVPESMQA, seg_pos={0 0}, upload=506107]: Uploading piece... Dec 29 16:48:57.000687620 (577h19m56.871770655s) [upload=506107]: Next returned nil segment Dec 29 16:48:57.000718509 (577h19m56.871801544s) [upload=506107]: Waiting for error group managing segments... Dec 29 16:48:57.300503856 (577h19m57.171586891s) [noise=true , address=145.79.227.39:4015 , piece_id=R2AXME6L4F5JERMAXDQZGYSK7EPPPP2KIMMSILT55FOH4TMAYZSQ, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.305556425 (577h19m57.17663947s) [noise=true , address=145.79.227.7:4295 , piece_id=F5BCBL7DIYDQHNZIDWOHYSVFCTD2XLLO3JQXBRGBZZRIMVPESMQA, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.312055471 (577h19m57.183138526s) [noise=true , address=145.79.227.50:4374 , piece_id=7CWHTWJ67GSASRNCMNHXTHDZDSSMNJORRRCGWB2JXCMSWQPMSX5Q, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.312894158 (577h19m57.183977203s) [noise=true , address=145.79.227.26:4655 , piece_id=ZF6O7GXLW5PMUAYRA4GAVQI3WY7TGTT6SUHVZ4CPN3MREGTB54LQ, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.313522775 (577h19m57.18460581s) [noise=true , address=145.79.227.38:4978 , piece_id=7DXGYJRU7EGSRQ2AKBH4MMSBWZGIY7BCTDKXUEANE4Z6FUBN2MXQ, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.321738509 (577h19m57.192821554s) [noise=true , address=145.79.227.48:4302 , piece_id=CFNGGADOG67ALTSWARZW75KTBLJYUMTGHO7C3WN3SRS4XTNOI2JA, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.327641048 (577h19m57.198724093s) [noise=true , address=145.79.227.51:4411 , piece_id=JIQSH4B4OE6EJAIAMWEZ2HESJS7RZLTR7H63F7FQZPINRESQF7RQ, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.329048979 (577h19m57.200132024s) [noise=true , address=145.79.227.3:4149 , piece_id=UZW6P4WWORFPOMJ6RVEVEZY73DI7XN5OJPOL3R4CYYARX426LR3Q, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.333101496 (577h19m57.204184531s) [noise=true , address=145.79.227.17:4972 , piece_id=DBZRK653HBWZSUA4FUZFCSPOROW7GCR4SZWOV4TRFDKSMI6PTZDQ, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.333573845 (577h19m57.204656891s) [noise=true , address=145.79.227.14:4839 , piece_id=TPKXYUKJSGOJ3J5YRAZI47NHBTDDAWQKIRR74LALWXZC2BUDYCCA, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.346069789 (577h19m57.217152824s) [noise=true , address=145.79.227.41:4086 , piece_id=HQGAPMW36KO6V6NMMJDS2GA63VG4HHBL5FIDXUEWK4IP42Z7QMIQ, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.886580652 (577h19m57.757663697s) [noise=true , address=45.91.116.11:4120 , piece_id=WJ4ZV4I3ITMKLEBFJY4MREKUIWX5PP5O2HJ5ZAXMH5VQ6ZGTEXMA, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.962262521 (577h19m57.833345556s) [noise=true , address=45.91.116.7:4012 , piece_id=W4BUUKSBJY4YQ277W3JATBTJ3A6BQZWFOTMLQRSCOR7G5NKG6ZJA, seg_pos={0 0}, upload=506107]: Done uploading piece. err: <nil> Dec 29 16:48:57.962282509 (577h19m57.833365544s) [seg_pos={0 0}, upload=506107]: Segment reached optimal threshold of 13 pieces. Dec 29 16:48:57.962596086 (577h19m57.833679141s) [noise=true , address=45.91.116.14:4191 , piece_id=5622LCYIVFSFQNUBD5N7DZPRQO3KHA7YUYHJ27QI2XWGCOQ7IVEQ, seg_pos={0 0}, upload=506107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:33370->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:33370->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33370->45.91.116.14:4191: use of closed network connection Dec 29 16:48:57.962611045 (577h19m57.83369409s) [noise=true , address=45.91.116.14:4191 , piece_id=5622LCYIVFSFQNUBD5N7DZPRQO3KHA7YUYHJ27QI2XWGCOQ7IVEQ, seg_pos={0 0}, upload=506107]: Long tail cancellation due to optimal threshold Dec 29 16:48:57.962651642 (577h19m57.833734677s) [noise=true , address=77.90.31.14:4663 , piece_id=W3XKR53ETYTXKZEEJEX4RYLYTMGJY473576QWJA3JOBCVYHQUTSA, seg_pos={0 0}, upload=506107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:56052->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:56052->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56052->77.90.31.14:4663: use of closed network connection Dec 29 16:48:57.962662292 (577h19m57.833745337s) [noise=true , address=77.90.31.14:4663 , piece_id=W3XKR53ETYTXKZEEJEX4RYLYTMGJY473576QWJA3JOBCVYHQUTSA, seg_pos={0 0}, upload=506107]: Long tail cancellation due to optimal threshold Dec 29 16:48:57.962766080 (577h19m57.833849115s) [noise=true , address=45.91.116.12:4228 , piece_id=NNWWHABCQNLVV5JMCENA5O4Y7KLZKTUP7T6QO7H3BJWUXSOBAD7A, seg_pos={0 0}, upload=506107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56062->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56062->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->45.91.116.12:4228: use of closed network connection Dec 29 16:48:57.962771600 (577h19m57.833854635s) [noise=true , address=45.91.116.12:4228 , piece_id=NNWWHABCQNLVV5JMCENA5O4Y7KLZKTUP7T6QO7H3BJWUXSOBAD7A, seg_pos={0 0}, upload=506107]: Long tail cancellation due to optimal threshold Dec 29 16:48:57.962867673 (577h19m57.833950718s) [seg_pos={0 0}, upload=506107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:48:57.962939420 (577h19m57.834022455s) [upload=506107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:48:57.962944980 (577h19m57.834028015s) [upload=506107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:48:58.254180726 (577h19m58.125263771s) [upload=506107]: Done uploading segments... Dec 29 16:48:58.254199522 (577h19m58.125282557s) [upload=506107]: Upload finished. err: <nil> Duration:* 30.003381027s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)