Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXXXX..X...] duration: 5.72353702s trace: https://jaeger.dev.storj.io/trace/1947da16f92feefb FAIL (2025-12-26 13:03:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q) context deadline exceeded ======LOGS====== Dec 26 13:01:21.143080601 (501h32m21.014163647s) [upload=440161]: Starting upload Dec 26 13:01:21.143156426 (501h32m21.014239471s) [upload=440161]: Done starting upload Dec 26 13:01:21.143250655 (501h32m21.0143337s) [upload=440161]: Uploading segments... Dec 26 13:01:21.143354954 (501h32m21.014437999s) [upload=440161]: Got next segment. Inline: false Dec 26 13:01:21.143430197 (501h32m21.014513242s) [upload=440161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:01:21.143436890 (501h32m21.014519925s) [upload=440161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:01:21.385752971 (501h32m21.256836026s) [seg_pos={0 0}, upload=440161]: Begin upload segment... Dec 26 13:01:21.385851770 (501h32m21.256934815s) [seg_pos={0 0}, upload=440161]: Done begin upload segment. Dec 26 13:01:21.385906434 (501h32m21.256989469s) [noise=true , address=77.90.31.12:4590 , piece_id=N63CFNEMZ3PDYYUAC7G2Q5TY2LHPZ6JXBZ6M6XQAA2KMWPQG4XSA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.385934107 (501h32m21.257017142s) [noise=true , address=77.90.31.15:4700 , piece_id=UW5C7VOHVR6KJYOQZBQGV7LLXT4EKEAYNII5NGXDQILURVKRY54Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.386594114 (501h32m21.257677149s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387021688 (501h32m21.258104723s) [noise=true , address=145.79.227.42:4123 , piece_id=THZIUV2CM4GWA74CQR4DO6FWCD5GH5ZMBJNJUSUWPCOGTPPCPCHQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387030164 (501h32m21.258113199s) [noise=true , address=145.79.227.3:4149 , piece_id=GVA7UDFTX6LXSRYAMROP77BX4VUF4X3QRIUJF3XW45YF342GEYFQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387321027 (501h32m21.258404072s) [noise=true , address=145.79.227.22:4511 , piece_id=TSNYCBKFWMTTBBG26MWEZC3RNNXLWF3RT74NAODCMFL43HVZG47Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387421749 (501h32m21.258504784s) [noise=true , address=45.91.116.5:4299 , piece_id=LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387704167 (501h32m21.258787202s) [noise=true , address=145.79.227.11:4730 , piece_id=BVBEY5DMASUWOC2NJR3WHBTIB2GY2UBXS5RXL5FQUNFEWGTYUZ4Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387782497 (501h32m21.258865542s) [noise=true , address=145.79.227.37:4566 , piece_id=RKAN35MQPPK72RJ7BSL3BN4H3LXPFAEKYMZLRAKST6R4N2V4GLRQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388000622 (501h32m21.259083667s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388211042 (501h32m21.259294077s) [noise=true , address=145.79.227.49:4337 , piece_id=R2XFLQKDAHIUHI3G2V7KG3YN7M7PBT3BQKWHW6GAJBOWWB55OO7A, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388276407 (501h32m21.259359442s) [noise=true , address=77.90.31.10:4520 , piece_id=BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388581268 (501h32m21.259664313s) [noise=true , address=145.79.227.15:4873 , piece_id=V4QEJ34MRBPPPCKWV6QTGSAILDGSOGQEJ6TRYOVBLCRU4KQBJJLA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388599482 (501h32m21.259682527s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388943868 (501h32m21.260026913s) [noise=true , address=45.91.116.13:4154 , piece_id=BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.389296940 (501h32m21.260379985s) [noise=true , address=45.91.116.4:4940 , piece_id=FV7HPVMKGOHFFPBBIVEV3PAIOJQKN5R4BAZ35XBQ7EMH6U2CT6ZQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:23.005073222 (501h32m22.876156287s) [upload=440161]: Next returned nil segment Dec 26 13:01:23.005092689 (501h32m22.876175754s) [upload=440161]: Waiting for error group managing segments... Dec 26 13:01:23.391475069 (501h32m23.262558104s) [noise=true , address=145.79.227.22:4511 , piece_id=TSNYCBKFWMTTBBG26MWEZC3RNNXLWF3RT74NAODCMFL43HVZG47Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.397493930 (501h32m23.268576975s) [noise=true , address=145.79.227.37:4566 , piece_id=RKAN35MQPPK72RJ7BSL3BN4H3LXPFAEKYMZLRAKST6R4N2V4GLRQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.409472429 (501h32m23.280555474s) [noise=true , address=145.79.227.42:4123 , piece_id=THZIUV2CM4GWA74CQR4DO6FWCD5GH5ZMBJNJUSUWPCOGTPPCPCHQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.409565316 (501h32m23.280648361s) [noise=true , address=145.79.227.49:4337 , piece_id=R2XFLQKDAHIUHI3G2V7KG3YN7M7PBT3BQKWHW6GAJBOWWB55OO7A, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.410237776 (501h32m23.281320811s) [noise=true , address=145.79.227.15:4873 , piece_id=V4QEJ34MRBPPPCKWV6QTGSAILDGSOGQEJ6TRYOVBLCRU4KQBJJLA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.416209728 (501h32m23.287292763s) [noise=true , address=145.79.227.11:4730 , piece_id=BVBEY5DMASUWOC2NJR3WHBTIB2GY2UBXS5RXL5FQUNFEWGTYUZ4Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.581095359 (501h32m23.452178404s) [noise=true , address=145.79.227.3:4149 , piece_id=GVA7UDFTX6LXSRYAMROP77BX4VUF4X3QRIUJF3XW45YF342GEYFQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.650109334 (501h32m23.521192369s) [noise=true , address=45.91.116.13:4154 , piece_id=BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.653225658 (501h32m23.524308693s) [noise=true , address=77.90.31.10:4520 , piece_id=BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.814186124 (501h32m23.685269169s) [noise=true , address=77.90.31.15:4700 , piece_id=UW5C7VOHVR6KJYOQZBQGV7LLXT4EKEAYNII5NGXDQILURVKRY54Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.849784570 (501h32m23.720867615s) [noise=true , address=45.91.116.4:4940 , piece_id=FV7HPVMKGOHFFPBBIVEV3PAIOJQKN5R4BAZ35XBQ7EMH6U2CT6ZQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.216343653 (501h32m24.087426688s) [noise=true , address=77.90.31.12:4590 , piece_id=N63CFNEMZ3PDYYUAC7G2Q5TY2LHPZ6JXBZ6M6XQAA2KMWPQG4XSA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.526872415 (501h32m24.39795546s) [noise=true , address=45.91.116.5:4299 , piece_id=LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.526894456 (501h32m24.397977491s) [seg_pos={0 0}, upload=440161]: Segment reached optimal threshold of 13 pieces. Dec 26 13:01:24.527057938 (501h32m24.398140973s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527069090 (501h32m24.398152135s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527256546 (501h32m24.398339581s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527264852 (501h32m24.398347897s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527387556 (501h32m24.398470591s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527393176 (501h32m24.398476212s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527412493 (501h32m24.398495538s) [seg_pos={0 0}, upload=440161]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:01:24.527524707 (501h32m24.398607742s) [upload=440161]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:01:24.527530398 (501h32m24.398613433s) [upload=440161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:01:24.810518988 (501h32m24.681602023s) [upload=440161]: Done uploading segments... Dec 26 13:01:24.810605021 (501h32m24.681688066s) [upload=440161]: Upload finished. err: <nil> Duration: 5.723570923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X....................X.XXX...X..X.............] duration: 31.492058407s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 13:03:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 31.4920858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..X..X....] duration: 3.858622164s trace: https://jaeger.dev.storj.io/trace/3d9bbbc5f19f2b7b FAIL (2025-12-26 13:03:02 UTC) orbiter warning: context deadline exceeded Duration: 3.858651479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X....................X.XXX...X..X............] duration: 48.705132541s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 13:02:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 48.705149104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXXXX..X..] duration: 5.70708467s trace: https://jaeger.dev.storj.io/trace/1947da16f92feefb FAIL (2025-12-26 13:02:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q) context deadline exceeded ======LOGS====== Dec 26 13:01:21.143080601 (501h32m21.014163647s) [upload=440161]: Starting upload Dec 26 13:01:21.143156426 (501h32m21.014239471s) [upload=440161]: Done starting upload Dec 26 13:01:21.143250655 (501h32m21.0143337s) [upload=440161]: Uploading segments... Dec 26 13:01:21.143354954 (501h32m21.014437999s) [upload=440161]: Got next segment. Inline: false Dec 26 13:01:21.143430197 (501h32m21.014513242s) [upload=440161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:01:21.143436890 (501h32m21.014519925s) [upload=440161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:01:21.385752971 (501h32m21.256836026s) [seg_pos={0 0}, upload=440161]: Begin upload segment... Dec 26 13:01:21.385851770 (501h32m21.256934815s) [seg_pos={0 0}, upload=440161]: Done begin upload segment. Dec 26 13:01:21.385906434 (501h32m21.256989469s) [noise=true , address=77.90.31.12:4590 , piece_id=N63CFNEMZ3PDYYUAC7G2Q5TY2LHPZ6JXBZ6M6XQAA2KMWPQG4XSA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.385934107 (501h32m21.257017142s) [noise=true , address=77.90.31.15:4700 , piece_id=UW5C7VOHVR6KJYOQZBQGV7LLXT4EKEAYNII5NGXDQILURVKRY54Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.386594114 (501h32m21.257677149s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387021688 (501h32m21.258104723s) [noise=true , address=145.79.227.42:4123 , piece_id=THZIUV2CM4GWA74CQR4DO6FWCD5GH5ZMBJNJUSUWPCOGTPPCPCHQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387030164 (501h32m21.258113199s) [noise=true , address=145.79.227.3:4149 , piece_id=GVA7UDFTX6LXSRYAMROP77BX4VUF4X3QRIUJF3XW45YF342GEYFQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387321027 (501h32m21.258404072s) [noise=true , address=145.79.227.22:4511 , piece_id=TSNYCBKFWMTTBBG26MWEZC3RNNXLWF3RT74NAODCMFL43HVZG47Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387421749 (501h32m21.258504784s) [noise=true , address=45.91.116.5:4299 , piece_id=LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387704167 (501h32m21.258787202s) [noise=true , address=145.79.227.11:4730 , piece_id=BVBEY5DMASUWOC2NJR3WHBTIB2GY2UBXS5RXL5FQUNFEWGTYUZ4Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387782497 (501h32m21.258865542s) [noise=true , address=145.79.227.37:4566 , piece_id=RKAN35MQPPK72RJ7BSL3BN4H3LXPFAEKYMZLRAKST6R4N2V4GLRQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388000622 (501h32m21.259083667s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388211042 (501h32m21.259294077s) [noise=true , address=145.79.227.49:4337 , piece_id=R2XFLQKDAHIUHI3G2V7KG3YN7M7PBT3BQKWHW6GAJBOWWB55OO7A, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388276407 (501h32m21.259359442s) [noise=true , address=77.90.31.10:4520 , piece_id=BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388581268 (501h32m21.259664313s) [noise=true , address=145.79.227.15:4873 , piece_id=V4QEJ34MRBPPPCKWV6QTGSAILDGSOGQEJ6TRYOVBLCRU4KQBJJLA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388599482 (501h32m21.259682527s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388943868 (501h32m21.260026913s) [noise=true , address=45.91.116.13:4154 , piece_id=BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.389296940 (501h32m21.260379985s) [noise=true , address=45.91.116.4:4940 , piece_id=FV7HPVMKGOHFFPBBIVEV3PAIOJQKN5R4BAZ35XBQ7EMH6U2CT6ZQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:23.005073222 (501h32m22.876156287s) [upload=440161]: Next returned nil segment Dec 26 13:01:23.005092689 (501h32m22.876175754s) [upload=440161]: Waiting for error group managing segments... Dec 26 13:01:23.391475069 (501h32m23.262558104s) [noise=true , address=145.79.227.22:4511 , piece_id=TSNYCBKFWMTTBBG26MWEZC3RNNXLWF3RT74NAODCMFL43HVZG47Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.397493930 (501h32m23.268576975s) [noise=true , address=145.79.227.37:4566 , piece_id=RKAN35MQPPK72RJ7BSL3BN4H3LXPFAEKYMZLRAKST6R4N2V4GLRQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.409472429 (501h32m23.280555474s) [noise=true , address=145.79.227.42:4123 , piece_id=THZIUV2CM4GWA74CQR4DO6FWCD5GH5ZMBJNJUSUWPCOGTPPCPCHQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.409565316 (501h32m23.280648361s) [noise=true , address=145.79.227.49:4337 , piece_id=R2XFLQKDAHIUHI3G2V7KG3YN7M7PBT3BQKWHW6GAJBOWWB55OO7A, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.410237776 (501h32m23.281320811s) [noise=true , address=145.79.227.15:4873 , piece_id=V4QEJ34MRBPPPCKWV6QTGSAILDGSOGQEJ6TRYOVBLCRU4KQBJJLA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.416209728 (501h32m23.287292763s) [noise=true , address=145.79.227.11:4730 , piece_id=BVBEY5DMASUWOC2NJR3WHBTIB2GY2UBXS5RXL5FQUNFEWGTYUZ4Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.581095359 (501h32m23.452178404s) [noise=true , address=145.79.227.3:4149 , piece_id=GVA7UDFTX6LXSRYAMROP77BX4VUF4X3QRIUJF3XW45YF342GEYFQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.650109334 (501h32m23.521192369s) [noise=true , address=45.91.116.13:4154 , piece_id=BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.653225658 (501h32m23.524308693s) [noise=true , address=77.90.31.10:4520 , piece_id=BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.814186124 (501h32m23.685269169s) [noise=true , address=77.90.31.15:4700 , piece_id=UW5C7VOHVR6KJYOQZBQGV7LLXT4EKEAYNII5NGXDQILURVKRY54Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.849784570 (501h32m23.720867615s) [noise=true , address=45.91.116.4:4940 , piece_id=FV7HPVMKGOHFFPBBIVEV3PAIOJQKN5R4BAZ35XBQ7EMH6U2CT6ZQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.216343653 (501h32m24.087426688s) [noise=true , address=77.90.31.12:4590 , piece_id=N63CFNEMZ3PDYYUAC7G2Q5TY2LHPZ6JXBZ6M6XQAA2KMWPQG4XSA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.526872415 (501h32m24.39795546s) [noise=true , address=45.91.116.5:4299 , piece_id=LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.526894456 (501h32m24.397977491s) [seg_pos={0 0}, upload=440161]: Segment reached optimal threshold of 13 pieces. Dec 26 13:01:24.527057938 (501h32m24.398140973s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527069090 (501h32m24.398152135s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527256546 (501h32m24.398339581s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527264852 (501h32m24.398347897s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527387556 (501h32m24.398470591s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527393176 (501h32m24.398476212s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527412493 (501h32m24.398495538s) [seg_pos={0 0}, upload=440161]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:01:24.527524707 (501h32m24.398607742s) [upload=440161]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:01:24.527530398 (501h32m24.398613433s) [upload=440161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:01:24.810518988 (501h32m24.681602023s) [upload=440161]: Done uploading segments... Dec 26 13:01:24.810605021 (501h32m24.681688066s) [upload=440161]: Upload finished. err: <nil> Duration: 5.707116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXXXX..X.] duration: 6.016337543s trace: https://jaeger.dev.storj.io/trace/1947da16f92feefb FAIL (2025-12-26 13:02:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q) context deadline exceeded ======LOGS====== Dec 26 13:01:21.143080601 (501h32m21.014163647s) [upload=440161]: Starting upload Dec 26 13:01:21.143156426 (501h32m21.014239471s) [upload=440161]: Done starting upload Dec 26 13:01:21.143250655 (501h32m21.0143337s) [upload=440161]: Uploading segments... Dec 26 13:01:21.143354954 (501h32m21.014437999s) [upload=440161]: Got next segment. Inline: false Dec 26 13:01:21.143430197 (501h32m21.014513242s) [upload=440161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:01:21.143436890 (501h32m21.014519925s) [upload=440161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:01:21.385752971 (501h32m21.256836026s) [seg_pos={0 0}, upload=440161]: Begin upload segment... Dec 26 13:01:21.385851770 (501h32m21.256934815s) [seg_pos={0 0}, upload=440161]: Done begin upload segment. Dec 26 13:01:21.385906434 (501h32m21.256989469s) [noise=true , address=77.90.31.12:4590 , piece_id=N63CFNEMZ3PDYYUAC7G2Q5TY2LHPZ6JXBZ6M6XQAA2KMWPQG4XSA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.385934107 (501h32m21.257017142s) [noise=true , address=77.90.31.15:4700 , piece_id=UW5C7VOHVR6KJYOQZBQGV7LLXT4EKEAYNII5NGXDQILURVKRY54Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.386594114 (501h32m21.257677149s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387021688 (501h32m21.258104723s) [noise=true , address=145.79.227.42:4123 , piece_id=THZIUV2CM4GWA74CQR4DO6FWCD5GH5ZMBJNJUSUWPCOGTPPCPCHQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387030164 (501h32m21.258113199s) [noise=true , address=145.79.227.3:4149 , piece_id=GVA7UDFTX6LXSRYAMROP77BX4VUF4X3QRIUJF3XW45YF342GEYFQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387321027 (501h32m21.258404072s) [noise=true , address=145.79.227.22:4511 , piece_id=TSNYCBKFWMTTBBG26MWEZC3RNNXLWF3RT74NAODCMFL43HVZG47Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387421749 (501h32m21.258504784s) [noise=true , address=45.91.116.5:4299 , piece_id=LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387704167 (501h32m21.258787202s) [noise=true , address=145.79.227.11:4730 , piece_id=BVBEY5DMASUWOC2NJR3WHBTIB2GY2UBXS5RXL5FQUNFEWGTYUZ4Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387782497 (501h32m21.258865542s) [noise=true , address=145.79.227.37:4566 , piece_id=RKAN35MQPPK72RJ7BSL3BN4H3LXPFAEKYMZLRAKST6R4N2V4GLRQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388000622 (501h32m21.259083667s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388211042 (501h32m21.259294077s) [noise=true , address=145.79.227.49:4337 , piece_id=R2XFLQKDAHIUHI3G2V7KG3YN7M7PBT3BQKWHW6GAJBOWWB55OO7A, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388276407 (501h32m21.259359442s) [noise=true , address=77.90.31.10:4520 , piece_id=BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388581268 (501h32m21.259664313s) [noise=true , address=145.79.227.15:4873 , piece_id=V4QEJ34MRBPPPCKWV6QTGSAILDGSOGQEJ6TRYOVBLCRU4KQBJJLA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388599482 (501h32m21.259682527s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388943868 (501h32m21.260026913s) [noise=true , address=45.91.116.13:4154 , piece_id=BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.389296940 (501h32m21.260379985s) [noise=true , address=45.91.116.4:4940 , piece_id=FV7HPVMKGOHFFPBBIVEV3PAIOJQKN5R4BAZ35XBQ7EMH6U2CT6ZQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:23.005073222 (501h32m22.876156287s) [upload=440161]: Next returned nil segment Dec 26 13:01:23.005092689 (501h32m22.876175754s) [upload=440161]: Waiting for error group managing segments... Dec 26 13:01:23.391475069 (501h32m23.262558104s) [noise=true , address=145.79.227.22:4511 , piece_id=TSNYCBKFWMTTBBG26MWEZC3RNNXLWF3RT74NAODCMFL43HVZG47Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.397493930 (501h32m23.268576975s) [noise=true , address=145.79.227.37:4566 , piece_id=RKAN35MQPPK72RJ7BSL3BN4H3LXPFAEKYMZLRAKST6R4N2V4GLRQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.409472429 (501h32m23.280555474s) [noise=true , address=145.79.227.42:4123 , piece_id=THZIUV2CM4GWA74CQR4DO6FWCD5GH5ZMBJNJUSUWPCOGTPPCPCHQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.409565316 (501h32m23.280648361s) [noise=true , address=145.79.227.49:4337 , piece_id=R2XFLQKDAHIUHI3G2V7KG3YN7M7PBT3BQKWHW6GAJBOWWB55OO7A, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.410237776 (501h32m23.281320811s) [noise=true , address=145.79.227.15:4873 , piece_id=V4QEJ34MRBPPPCKWV6QTGSAILDGSOGQEJ6TRYOVBLCRU4KQBJJLA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.416209728 (501h32m23.287292763s) [noise=true , address=145.79.227.11:4730 , piece_id=BVBEY5DMASUWOC2NJR3WHBTIB2GY2UBXS5RXL5FQUNFEWGTYUZ4Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.581095359 (501h32m23.452178404s) [noise=true , address=145.79.227.3:4149 , piece_id=GVA7UDFTX6LXSRYAMROP77BX4VUF4X3QRIUJF3XW45YF342GEYFQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.650109334 (501h32m23.521192369s) [noise=true , address=45.91.116.13:4154 , piece_id=BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.653225658 (501h32m23.524308693s) [noise=true , address=77.90.31.10:4520 , piece_id=BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.814186124 (501h32m23.685269169s) [noise=true , address=77.90.31.15:4700 , piece_id=UW5C7VOHVR6KJYOQZBQGV7LLXT4EKEAYNII5NGXDQILURVKRY54Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.849784570 (501h32m23.720867615s) [noise=true , address=45.91.116.4:4940 , piece_id=FV7HPVMKGOHFFPBBIVEV3PAIOJQKN5R4BAZ35XBQ7EMH6U2CT6ZQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.216343653 (501h32m24.087426688s) [noise=true , address=77.90.31.12:4590 , piece_id=N63CFNEMZ3PDYYUAC7G2Q5TY2LHPZ6JXBZ6M6XQAA2KMWPQG4XSA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.526872415 (501h32m24.39795546s) [noise=true , address=45.91.116.5:4299 , piece_id=LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.526894456 (501h32m24.397977491s) [seg_pos={0 0}, upload=440161]: Segment reached optimal threshold of 13 pieces. Dec 26 13:01:24.527057938 (501h32m24.398140973s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527069090 (501h32m24.398152135s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527256546 (501h32m24.398339581s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527264852 (501h32m24.398347897s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527387556 (501h32m24.398470591s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527393176 (501h32m24.398476212s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527412493 (501h32m24.398495538s) [seg_pos={0 0}, upload=440161]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:01:24.527524707 (501h32m24.398607742s) [upload=440161]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:01:24.527530398 (501h32m24.398613433s) [upload=440161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:01:24.810518988 (501h32m24.681602023s) [upload=440161]: Done uploading segments... Dec 26 13:01:24.810605021 (501h32m24.681688066s) [upload=440161]: Upload finished. err: <nil> Duration: 6.016366227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..X..X...] duration: 4.233412571s trace: https://jaeger.dev.storj.io/trace/3d9bbbc5f19f2b7b FAIL (2025-12-26 13:02:11 UTC) orbiter warning: context deadline exceeded Duration: 4.233444131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXXXX..X] duration: 30.00559461s trace: https://jaeger.dev.storj.io/trace/1947da16f92feefb FAIL (2025-12-26 13:01:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q) context deadline exceeded ======LOGS====== Dec 26 13:01:21.143080601 (501h32m21.014163647s) [upload=440161]: Starting upload Dec 26 13:01:21.143156426 (501h32m21.014239471s) [upload=440161]: Done starting upload Dec 26 13:01:21.143250655 (501h32m21.0143337s) [upload=440161]: Uploading segments... Dec 26 13:01:21.143354954 (501h32m21.014437999s) [upload=440161]: Got next segment. Inline: false Dec 26 13:01:21.143430197 (501h32m21.014513242s) [upload=440161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:01:21.143436890 (501h32m21.014519925s) [upload=440161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:01:21.385752971 (501h32m21.256836026s) [seg_pos={0 0}, upload=440161]: Begin upload segment... Dec 26 13:01:21.385851770 (501h32m21.256934815s) [seg_pos={0 0}, upload=440161]: Done begin upload segment. Dec 26 13:01:21.385906434 (501h32m21.256989469s) [noise=true , address=77.90.31.12:4590 , piece_id=N63CFNEMZ3PDYYUAC7G2Q5TY2LHPZ6JXBZ6M6XQAA2KMWPQG4XSA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.385934107 (501h32m21.257017142s) [noise=true , address=77.90.31.15:4700 , piece_id=UW5C7VOHVR6KJYOQZBQGV7LLXT4EKEAYNII5NGXDQILURVKRY54Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.386594114 (501h32m21.257677149s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387021688 (501h32m21.258104723s) [noise=true , address=145.79.227.42:4123 , piece_id=THZIUV2CM4GWA74CQR4DO6FWCD5GH5ZMBJNJUSUWPCOGTPPCPCHQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387030164 (501h32m21.258113199s) [noise=true , address=145.79.227.3:4149 , piece_id=GVA7UDFTX6LXSRYAMROP77BX4VUF4X3QRIUJF3XW45YF342GEYFQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387321027 (501h32m21.258404072s) [noise=true , address=145.79.227.22:4511 , piece_id=TSNYCBKFWMTTBBG26MWEZC3RNNXLWF3RT74NAODCMFL43HVZG47Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387421749 (501h32m21.258504784s) [noise=true , address=45.91.116.5:4299 , piece_id=LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387704167 (501h32m21.258787202s) [noise=true , address=145.79.227.11:4730 , piece_id=BVBEY5DMASUWOC2NJR3WHBTIB2GY2UBXS5RXL5FQUNFEWGTYUZ4Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.387782497 (501h32m21.258865542s) [noise=true , address=145.79.227.37:4566 , piece_id=RKAN35MQPPK72RJ7BSL3BN4H3LXPFAEKYMZLRAKST6R4N2V4GLRQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388000622 (501h32m21.259083667s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388211042 (501h32m21.259294077s) [noise=true , address=145.79.227.49:4337 , piece_id=R2XFLQKDAHIUHI3G2V7KG3YN7M7PBT3BQKWHW6GAJBOWWB55OO7A, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388276407 (501h32m21.259359442s) [noise=true , address=77.90.31.10:4520 , piece_id=BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388581268 (501h32m21.259664313s) [noise=true , address=145.79.227.15:4873 , piece_id=V4QEJ34MRBPPPCKWV6QTGSAILDGSOGQEJ6TRYOVBLCRU4KQBJJLA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388599482 (501h32m21.259682527s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.388943868 (501h32m21.260026913s) [noise=true , address=45.91.116.13:4154 , piece_id=BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:21.389296940 (501h32m21.260379985s) [noise=true , address=45.91.116.4:4940 , piece_id=FV7HPVMKGOHFFPBBIVEV3PAIOJQKN5R4BAZ35XBQ7EMH6U2CT6ZQ, seg_pos={0 0}, upload=440161]: Uploading piece... Dec 26 13:01:23.005073222 (501h32m22.876156287s) [upload=440161]: Next returned nil segment Dec 26 13:01:23.005092689 (501h32m22.876175754s) [upload=440161]: Waiting for error group managing segments... Dec 26 13:01:23.391475069 (501h32m23.262558104s) [noise=true , address=145.79.227.22:4511 , piece_id=TSNYCBKFWMTTBBG26MWEZC3RNNXLWF3RT74NAODCMFL43HVZG47Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.397493930 (501h32m23.268576975s) [noise=true , address=145.79.227.37:4566 , piece_id=RKAN35MQPPK72RJ7BSL3BN4H3LXPFAEKYMZLRAKST6R4N2V4GLRQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.409472429 (501h32m23.280555474s) [noise=true , address=145.79.227.42:4123 , piece_id=THZIUV2CM4GWA74CQR4DO6FWCD5GH5ZMBJNJUSUWPCOGTPPCPCHQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.409565316 (501h32m23.280648361s) [noise=true , address=145.79.227.49:4337 , piece_id=R2XFLQKDAHIUHI3G2V7KG3YN7M7PBT3BQKWHW6GAJBOWWB55OO7A, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.410237776 (501h32m23.281320811s) [noise=true , address=145.79.227.15:4873 , piece_id=V4QEJ34MRBPPPCKWV6QTGSAILDGSOGQEJ6TRYOVBLCRU4KQBJJLA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.416209728 (501h32m23.287292763s) [noise=true , address=145.79.227.11:4730 , piece_id=BVBEY5DMASUWOC2NJR3WHBTIB2GY2UBXS5RXL5FQUNFEWGTYUZ4Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.581095359 (501h32m23.452178404s) [noise=true , address=145.79.227.3:4149 , piece_id=GVA7UDFTX6LXSRYAMROP77BX4VUF4X3QRIUJF3XW45YF342GEYFQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.650109334 (501h32m23.521192369s) [noise=true , address=45.91.116.13:4154 , piece_id=BGPKDVEITBOV2AHCVVD2Q6DNC67CAYZC2B7V66KRMQQ4FB4LHF4Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.653225658 (501h32m23.524308693s) [noise=true , address=77.90.31.10:4520 , piece_id=BPX65UHTYZ3OHIV7D7UU4JYK3VN7BKT2TJ7RS7HRVKHAQATO2QKA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.814186124 (501h32m23.685269169s) [noise=true , address=77.90.31.15:4700 , piece_id=UW5C7VOHVR6KJYOQZBQGV7LLXT4EKEAYNII5NGXDQILURVKRY54Q, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:23.849784570 (501h32m23.720867615s) [noise=true , address=45.91.116.4:4940 , piece_id=FV7HPVMKGOHFFPBBIVEV3PAIOJQKN5R4BAZ35XBQ7EMH6U2CT6ZQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.216343653 (501h32m24.087426688s) [noise=true , address=77.90.31.12:4590 , piece_id=N63CFNEMZ3PDYYUAC7G2Q5TY2LHPZ6JXBZ6M6XQAA2KMWPQG4XSA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.526872415 (501h32m24.39795546s) [noise=true , address=45.91.116.5:4299 , piece_id=LHIXYFRATOQDEP42DELEFFJSHA26UP4MJE4UAG3OLL2M5FBDSRLQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: <nil> Dec 26 13:01:24.526894456 (501h32m24.397977491s) [seg_pos={0 0}, upload=440161]: Segment reached optimal threshold of 13 pieces. Dec 26 13:01:24.527057938 (501h32m24.398140973s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527069090 (501h32m24.398152135s) [noise=true , address=145.79.227.40:4050 , piece_id=STTGD3NAGEM4EVOCZ2YBHP6SDC4CPE73PM22WQ4RDGXBL74YQRIQ, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527256546 (501h32m24.398339581s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527264852 (501h32m24.398347897s) [noise=true , address=45.91.116.12:4225 , piece_id=EJIHH23GRQADV2K6PR6FI7XI5ELN3A4SIPOJ5EMNZR2GZT5WOMKA, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527387556 (501h32m24.398470591s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 13:01:24.527393176 (501h32m24.398476212s) [noise=true , address=45.91.116.11:4120 , piece_id=AZOGVM6CHOKLLJPXECIEN7X3LS2BGVYONDUDQX6MRFDQF3SPASOA, seg_pos={0 0}, upload=440161]: Long tail cancellation due to optimal threshold Dec 26 13:01:24.527412493 (501h32m24.398495538s) [seg_pos={0 0}, upload=440161]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:01:24.527524707 (501h32m24.398607742s) [upload=440161]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:01:24.527530398 (501h32m24.398613433s) [upload=440161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:01:24.810518988 (501h32m24.681602023s) [upload=440161]: Done uploading segments... Dec 26 13:01:24.810605021 (501h32m24.681688066s) [upload=440161]: Upload finished. err: <nil> Duration: 30.005616081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X....................X.XXX...X..X...........] duration: 51.842265485s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 13:01:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 51.842280974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..X..X..] duration: 4.130518772s trace: https://jaeger.dev.storj.io/trace/3d9bbbc5f19f2b7b FAIL (2025-12-26 13:01:49 UTC) orbiter warning: context deadline exceeded Duration: 4.130557838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..X..X.] duration: 4.840943266s trace: https://jaeger.dev.storj.io/trace/3d9bbbc5f19f2b7b FAIL (2025-12-26 13:01:02 UTC) orbiter warning: context deadline exceeded Duration: 4.841019962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X....................X.XXX...X..X..........] duration: 48.224943867s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 13:00:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 48.224974825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXXXX..] duration: 5.80484734s trace: https://jaeger.dev.storj.io/trace/62f0cf20d23e4dcb FAIL (2025-12-26 13:00:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA) context deadline exceeded ======LOGS====== Dec 26 12:59:24.231230998 (501h30m24.102314043s) [upload=440131]: Starting upload Dec 26 12:59:24.231393917 (501h30m24.102476962s) [upload=440131]: Done starting upload Dec 26 12:59:24.231480132 (501h30m24.102563167s) [upload=440131]: Uploading segments... Dec 26 12:59:24.231757630 (501h30m24.102840666s) [upload=440131]: Got next segment. Inline: false Dec 26 12:59:24.231800622 (501h30m24.102883657s) [upload=440131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:59:24.231806203 (501h30m24.102889238s) [upload=440131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:59:24.476512768 (501h30m24.347595803s) [seg_pos={0 0}, upload=440131]: Begin upload segment... Dec 26 12:59:24.476621885 (501h30m24.34770492s) [seg_pos={0 0}, upload=440131]: Done begin upload segment. Dec 26 12:59:24.476699964 (501h30m24.347782999s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.476790146 (501h30m24.347873181s) [noise=true , address=145.79.227.13:4804 , piece_id=3N6LJSS3IF42W6YJGCX2JPSVDKZ63JJHVMQX5KOD2UWJXDOZYTZQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.478636832 (501h30m24.349719867s) [noise=true , address=145.79.227.38:4977 , piece_id=FZVDQHYMY7QBS2QJB7GSBAYBKMJGDB3KAD7F43P5WIQ7YTLWFHCQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.478994112 (501h30m24.350077147s) [noise=true , address=45.91.116.5:4297 , piece_id=J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.479291188 (501h30m24.350374223s) [noise=true , address=45.91.116.4:4937 , piece_id=T2LAZCFPAQKVVN7UKLGBKFNPPHSMVUNP35QGA3P7BUTHCDV2SHIA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.479698824 (501h30m24.350781859s) [noise=true , address=145.79.227.39:4014 , piece_id=KTEPYRY4UQRRZFXPE3NQ5O3UYAVZMU5KBPAXPJNC3BBMCQCMZWPA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.480019395 (501h30m24.35110243s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.480356917 (501h30m24.351439952s) [noise=true , address=145.79.227.52:4446 , piece_id=KHTOBTC5BYO6XP4D6LWRDDLIH6ZW5XI7JYQAC7Z7GPVCMCDJT3UQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.481407297 (501h30m24.352490332s) [noise=true , address=45.91.116.13:4154 , piece_id=HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.481591007 (501h30m24.352674053s) [noise=true , address=145.79.227.2:4115 , piece_id=APV47H6ZBSBQGPSZDNG6EXE3KUOFHR6BNNBDWH7NXRJFBUJWVYGQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.483326462 (501h30m24.354409497s) [noise=true , address=145.79.227.41:4086 , piece_id=SCFWUXVZT7F64FLO6ODQRCKUVJQ4WHT3F7PHBV2HP3JFEO5PC74Q, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.485057738 (501h30m24.356140773s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.487270562 (501h30m24.358353607s) [noise=true , address=145.79.227.45:4231 , piece_id=J3VJJ34DMXNU5U2AYURPXZMTMW5LRMM5JJN4UN3NP2YGZ67J5MOA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.488511676 (501h30m24.359594711s) [noise=true , address=145.79.227.21:4475 , piece_id=GM5WX4BAJGMFHIKCCTQ2EI2FEHBONWNRI4CGJCG4CK5Z6Y5TTZGQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.491513291 (501h30m24.362596326s) [noise=true , address=145.79.227.42:4122 , piece_id=M6VMD2ZL4CBWJD3DMZIG4GJB6JKVI27O3NZCQOFYLB6AFFYWJPYA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.492999410 (501h30m24.364082445s) [noise=true , address=145.79.227.4:4185 , piece_id=KSKYUGW4ZYRA2QONDGVI2FKCO26DSGW2WCWWVSX2LVBQV3TTEYZA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:26.189315254 (501h30m26.060398289s) [upload=440131]: Next returned nil segment Dec 26 12:59:26.189364188 (501h30m26.060447223s) [upload=440131]: Waiting for error group managing segments... Dec 26 12:59:26.479418801 (501h30m26.350501846s) [noise=true , address=145.79.227.39:4014 , piece_id=KTEPYRY4UQRRZFXPE3NQ5O3UYAVZMU5KBPAXPJNC3BBMCQCMZWPA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.484248899 (501h30m26.355331944s) [noise=true , address=145.79.227.38:4977 , piece_id=FZVDQHYMY7QBS2QJB7GSBAYBKMJGDB3KAD7F43P5WIQ7YTLWFHCQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.484605708 (501h30m26.355688743s) [noise=true , address=145.79.227.2:4115 , piece_id=APV47H6ZBSBQGPSZDNG6EXE3KUOFHR6BNNBDWH7NXRJFBUJWVYGQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.494981425 (501h30m26.366064461s) [noise=true , address=145.79.227.21:4475 , piece_id=GM5WX4BAJGMFHIKCCTQ2EI2FEHBONWNRI4CGJCG4CK5Z6Y5TTZGQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.499068679 (501h30m26.370151724s) [noise=true , address=145.79.227.4:4185 , piece_id=KSKYUGW4ZYRA2QONDGVI2FKCO26DSGW2WCWWVSX2LVBQV3TTEYZA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.499092564 (501h30m26.370175609s) [noise=true , address=145.79.227.45:4231 , piece_id=J3VJJ34DMXNU5U2AYURPXZMTMW5LRMM5JJN4UN3NP2YGZ67J5MOA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.503937500 (501h30m26.375020545s) [noise=true , address=145.79.227.52:4446 , piece_id=KHTOBTC5BYO6XP4D6LWRDDLIH6ZW5XI7JYQAC7Z7GPVCMCDJT3UQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.504793870 (501h30m26.375876905s) [noise=true , address=145.79.227.42:4122 , piece_id=M6VMD2ZL4CBWJD3DMZIG4GJB6JKVI27O3NZCQOFYLB6AFFYWJPYA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.510485258 (501h30m26.381568293s) [noise=true , address=145.79.227.41:4086 , piece_id=SCFWUXVZT7F64FLO6ODQRCKUVJQ4WHT3F7PHBV2HP3JFEO5PC74Q, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.516462209 (501h30m26.387545244s) [noise=true , address=145.79.227.13:4804 , piece_id=3N6LJSS3IF42W6YJGCX2JPSVDKZ63JJHVMQX5KOD2UWJXDOZYTZQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.704044719 (501h30m26.575127754s) [noise=true , address=45.91.116.5:4297 , piece_id=J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.726190408 (501h30m26.597273453s) [noise=true , address=45.91.116.4:4937 , piece_id=T2LAZCFPAQKVVN7UKLGBKFNPPHSMVUNP35QGA3P7BUTHCDV2SHIA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.836341746 (501h30m26.707424791s) [noise=true , address=45.91.116.13:4154 , piece_id=HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.836365040 (501h30m26.707448085s) [seg_pos={0 0}, upload=440131]: Segment reached optimal threshold of 13 pieces. Dec 26 12:59:26.836601220 (501h30m26.707684255s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836617752 (501h30m26.707700787s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836631598 (501h30m26.707714643s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836639072 (501h30m26.707722107s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836783136 (501h30m26.707866181s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836789809 (501h30m26.707872854s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836805469 (501h30m26.707888504s) [seg_pos={0 0}, upload=440131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:59:26.836909266 (501h30m26.707992301s) [upload=440131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:59:26.836914466 (501h30m26.707997501s) [upload=440131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:59:27.123356883 (501h30m26.994439948s) [upload=440131]: Done uploading segments... Dec 26 12:59:27.123415144 (501h30m26.994498199s) [upload=440131]: Upload finished. err: <nil> Duration: 5.804870713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.....XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXXXX.] duration: 6.028004478s trace: https://jaeger.dev.storj.io/trace/62f0cf20d23e4dcb FAIL (2025-12-26 13:00:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA) context deadline exceeded ======LOGS====== Dec 26 12:59:24.231230998 (501h30m24.102314043s) [upload=440131]: Starting upload Dec 26 12:59:24.231393917 (501h30m24.102476962s) [upload=440131]: Done starting upload Dec 26 12:59:24.231480132 (501h30m24.102563167s) [upload=440131]: Uploading segments... Dec 26 12:59:24.231757630 (501h30m24.102840666s) [upload=440131]: Got next segment. Inline: false Dec 26 12:59:24.231800622 (501h30m24.102883657s) [upload=440131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:59:24.231806203 (501h30m24.102889238s) [upload=440131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:59:24.476512768 (501h30m24.347595803s) [seg_pos={0 0}, upload=440131]: Begin upload segment... Dec 26 12:59:24.476621885 (501h30m24.34770492s) [seg_pos={0 0}, upload=440131]: Done begin upload segment. Dec 26 12:59:24.476699964 (501h30m24.347782999s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.476790146 (501h30m24.347873181s) [noise=true , address=145.79.227.13:4804 , piece_id=3N6LJSS3IF42W6YJGCX2JPSVDKZ63JJHVMQX5KOD2UWJXDOZYTZQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.478636832 (501h30m24.349719867s) [noise=true , address=145.79.227.38:4977 , piece_id=FZVDQHYMY7QBS2QJB7GSBAYBKMJGDB3KAD7F43P5WIQ7YTLWFHCQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.478994112 (501h30m24.350077147s) [noise=true , address=45.91.116.5:4297 , piece_id=J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.479291188 (501h30m24.350374223s) [noise=true , address=45.91.116.4:4937 , piece_id=T2LAZCFPAQKVVN7UKLGBKFNPPHSMVUNP35QGA3P7BUTHCDV2SHIA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.479698824 (501h30m24.350781859s) [noise=true , address=145.79.227.39:4014 , piece_id=KTEPYRY4UQRRZFXPE3NQ5O3UYAVZMU5KBPAXPJNC3BBMCQCMZWPA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.480019395 (501h30m24.35110243s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.480356917 (501h30m24.351439952s) [noise=true , address=145.79.227.52:4446 , piece_id=KHTOBTC5BYO6XP4D6LWRDDLIH6ZW5XI7JYQAC7Z7GPVCMCDJT3UQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.481407297 (501h30m24.352490332s) [noise=true , address=45.91.116.13:4154 , piece_id=HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.481591007 (501h30m24.352674053s) [noise=true , address=145.79.227.2:4115 , piece_id=APV47H6ZBSBQGPSZDNG6EXE3KUOFHR6BNNBDWH7NXRJFBUJWVYGQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.483326462 (501h30m24.354409497s) [noise=true , address=145.79.227.41:4086 , piece_id=SCFWUXVZT7F64FLO6ODQRCKUVJQ4WHT3F7PHBV2HP3JFEO5PC74Q, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.485057738 (501h30m24.356140773s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.487270562 (501h30m24.358353607s) [noise=true , address=145.79.227.45:4231 , piece_id=J3VJJ34DMXNU5U2AYURPXZMTMW5LRMM5JJN4UN3NP2YGZ67J5MOA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.488511676 (501h30m24.359594711s) [noise=true , address=145.79.227.21:4475 , piece_id=GM5WX4BAJGMFHIKCCTQ2EI2FEHBONWNRI4CGJCG4CK5Z6Y5TTZGQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.491513291 (501h30m24.362596326s) [noise=true , address=145.79.227.42:4122 , piece_id=M6VMD2ZL4CBWJD3DMZIG4GJB6JKVI27O3NZCQOFYLB6AFFYWJPYA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.492999410 (501h30m24.364082445s) [noise=true , address=145.79.227.4:4185 , piece_id=KSKYUGW4ZYRA2QONDGVI2FKCO26DSGW2WCWWVSX2LVBQV3TTEYZA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:26.189315254 (501h30m26.060398289s) [upload=440131]: Next returned nil segment Dec 26 12:59:26.189364188 (501h30m26.060447223s) [upload=440131]: Waiting for error group managing segments... Dec 26 12:59:26.479418801 (501h30m26.350501846s) [noise=true , address=145.79.227.39:4014 , piece_id=KTEPYRY4UQRRZFXPE3NQ5O3UYAVZMU5KBPAXPJNC3BBMCQCMZWPA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.484248899 (501h30m26.355331944s) [noise=true , address=145.79.227.38:4977 , piece_id=FZVDQHYMY7QBS2QJB7GSBAYBKMJGDB3KAD7F43P5WIQ7YTLWFHCQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.484605708 (501h30m26.355688743s) [noise=true , address=145.79.227.2:4115 , piece_id=APV47H6ZBSBQGPSZDNG6EXE3KUOFHR6BNNBDWH7NXRJFBUJWVYGQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.494981425 (501h30m26.366064461s) [noise=true , address=145.79.227.21:4475 , piece_id=GM5WX4BAJGMFHIKCCTQ2EI2FEHBONWNRI4CGJCG4CK5Z6Y5TTZGQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.499068679 (501h30m26.370151724s) [noise=true , address=145.79.227.4:4185 , piece_id=KSKYUGW4ZYRA2QONDGVI2FKCO26DSGW2WCWWVSX2LVBQV3TTEYZA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.499092564 (501h30m26.370175609s) [noise=true , address=145.79.227.45:4231 , piece_id=J3VJJ34DMXNU5U2AYURPXZMTMW5LRMM5JJN4UN3NP2YGZ67J5MOA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.503937500 (501h30m26.375020545s) [noise=true , address=145.79.227.52:4446 , piece_id=KHTOBTC5BYO6XP4D6LWRDDLIH6ZW5XI7JYQAC7Z7GPVCMCDJT3UQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.504793870 (501h30m26.375876905s) [noise=true , address=145.79.227.42:4122 , piece_id=M6VMD2ZL4CBWJD3DMZIG4GJB6JKVI27O3NZCQOFYLB6AFFYWJPYA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.510485258 (501h30m26.381568293s) [noise=true , address=145.79.227.41:4086 , piece_id=SCFWUXVZT7F64FLO6ODQRCKUVJQ4WHT3F7PHBV2HP3JFEO5PC74Q, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.516462209 (501h30m26.387545244s) [noise=true , address=145.79.227.13:4804 , piece_id=3N6LJSS3IF42W6YJGCX2JPSVDKZ63JJHVMQX5KOD2UWJXDOZYTZQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.704044719 (501h30m26.575127754s) [noise=true , address=45.91.116.5:4297 , piece_id=J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.726190408 (501h30m26.597273453s) [noise=true , address=45.91.116.4:4937 , piece_id=T2LAZCFPAQKVVN7UKLGBKFNPPHSMVUNP35QGA3P7BUTHCDV2SHIA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.836341746 (501h30m26.707424791s) [noise=true , address=45.91.116.13:4154 , piece_id=HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.836365040 (501h30m26.707448085s) [seg_pos={0 0}, upload=440131]: Segment reached optimal threshold of 13 pieces. Dec 26 12:59:26.836601220 (501h30m26.707684255s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836617752 (501h30m26.707700787s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836631598 (501h30m26.707714643s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836639072 (501h30m26.707722107s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836783136 (501h30m26.707866181s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836789809 (501h30m26.707872854s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836805469 (501h30m26.707888504s) [seg_pos={0 0}, upload=440131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:59:26.836909266 (501h30m26.707992301s) [upload=440131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:59:26.836914466 (501h30m26.707997501s) [upload=440131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:59:27.123356883 (501h30m26.994439948s) [upload=440131]: Done uploading segments... Dec 26 12:59:27.123415144 (501h30m26.994498199s) [upload=440131]: Upload finished. err: <nil> Duration: 6.028031438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................X....................X.XXX...X..X.........] duration: 39.901047609s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 13:00:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 39.901069251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..X..X] duration: 5.001614422s trace: https://jaeger.dev.storj.io/trace/3d9bbbc5f19f2b7b FAIL (2025-12-26 12:59:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001637074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXXXX] duration: 30.003386084s trace: https://jaeger.dev.storj.io/trace/62f0cf20d23e4dcb FAIL (2025-12-26 12:59:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA) context deadline exceeded ======LOGS====== Dec 26 12:59:24.231230998 (501h30m24.102314043s) [upload=440131]: Starting upload Dec 26 12:59:24.231393917 (501h30m24.102476962s) [upload=440131]: Done starting upload Dec 26 12:59:24.231480132 (501h30m24.102563167s) [upload=440131]: Uploading segments... Dec 26 12:59:24.231757630 (501h30m24.102840666s) [upload=440131]: Got next segment. Inline: false Dec 26 12:59:24.231800622 (501h30m24.102883657s) [upload=440131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:59:24.231806203 (501h30m24.102889238s) [upload=440131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:59:24.476512768 (501h30m24.347595803s) [seg_pos={0 0}, upload=440131]: Begin upload segment... Dec 26 12:59:24.476621885 (501h30m24.34770492s) [seg_pos={0 0}, upload=440131]: Done begin upload segment. Dec 26 12:59:24.476699964 (501h30m24.347782999s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.476790146 (501h30m24.347873181s) [noise=true , address=145.79.227.13:4804 , piece_id=3N6LJSS3IF42W6YJGCX2JPSVDKZ63JJHVMQX5KOD2UWJXDOZYTZQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.478636832 (501h30m24.349719867s) [noise=true , address=145.79.227.38:4977 , piece_id=FZVDQHYMY7QBS2QJB7GSBAYBKMJGDB3KAD7F43P5WIQ7YTLWFHCQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.478994112 (501h30m24.350077147s) [noise=true , address=45.91.116.5:4297 , piece_id=J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.479291188 (501h30m24.350374223s) [noise=true , address=45.91.116.4:4937 , piece_id=T2LAZCFPAQKVVN7UKLGBKFNPPHSMVUNP35QGA3P7BUTHCDV2SHIA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.479698824 (501h30m24.350781859s) [noise=true , address=145.79.227.39:4014 , piece_id=KTEPYRY4UQRRZFXPE3NQ5O3UYAVZMU5KBPAXPJNC3BBMCQCMZWPA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.480019395 (501h30m24.35110243s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.480356917 (501h30m24.351439952s) [noise=true , address=145.79.227.52:4446 , piece_id=KHTOBTC5BYO6XP4D6LWRDDLIH6ZW5XI7JYQAC7Z7GPVCMCDJT3UQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.481407297 (501h30m24.352490332s) [noise=true , address=45.91.116.13:4154 , piece_id=HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.481591007 (501h30m24.352674053s) [noise=true , address=145.79.227.2:4115 , piece_id=APV47H6ZBSBQGPSZDNG6EXE3KUOFHR6BNNBDWH7NXRJFBUJWVYGQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.483326462 (501h30m24.354409497s) [noise=true , address=145.79.227.41:4086 , piece_id=SCFWUXVZT7F64FLO6ODQRCKUVJQ4WHT3F7PHBV2HP3JFEO5PC74Q, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.485057738 (501h30m24.356140773s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.487270562 (501h30m24.358353607s) [noise=true , address=145.79.227.45:4231 , piece_id=J3VJJ34DMXNU5U2AYURPXZMTMW5LRMM5JJN4UN3NP2YGZ67J5MOA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.488511676 (501h30m24.359594711s) [noise=true , address=145.79.227.21:4475 , piece_id=GM5WX4BAJGMFHIKCCTQ2EI2FEHBONWNRI4CGJCG4CK5Z6Y5TTZGQ, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.491513291 (501h30m24.362596326s) [noise=true , address=145.79.227.42:4122 , piece_id=M6VMD2ZL4CBWJD3DMZIG4GJB6JKVI27O3NZCQOFYLB6AFFYWJPYA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:24.492999410 (501h30m24.364082445s) [noise=true , address=145.79.227.4:4185 , piece_id=KSKYUGW4ZYRA2QONDGVI2FKCO26DSGW2WCWWVSX2LVBQV3TTEYZA, seg_pos={0 0}, upload=440131]: Uploading piece... Dec 26 12:59:26.189315254 (501h30m26.060398289s) [upload=440131]: Next returned nil segment Dec 26 12:59:26.189364188 (501h30m26.060447223s) [upload=440131]: Waiting for error group managing segments... Dec 26 12:59:26.479418801 (501h30m26.350501846s) [noise=true , address=145.79.227.39:4014 , piece_id=KTEPYRY4UQRRZFXPE3NQ5O3UYAVZMU5KBPAXPJNC3BBMCQCMZWPA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.484248899 (501h30m26.355331944s) [noise=true , address=145.79.227.38:4977 , piece_id=FZVDQHYMY7QBS2QJB7GSBAYBKMJGDB3KAD7F43P5WIQ7YTLWFHCQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.484605708 (501h30m26.355688743s) [noise=true , address=145.79.227.2:4115 , piece_id=APV47H6ZBSBQGPSZDNG6EXE3KUOFHR6BNNBDWH7NXRJFBUJWVYGQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.494981425 (501h30m26.366064461s) [noise=true , address=145.79.227.21:4475 , piece_id=GM5WX4BAJGMFHIKCCTQ2EI2FEHBONWNRI4CGJCG4CK5Z6Y5TTZGQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.499068679 (501h30m26.370151724s) [noise=true , address=145.79.227.4:4185 , piece_id=KSKYUGW4ZYRA2QONDGVI2FKCO26DSGW2WCWWVSX2LVBQV3TTEYZA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.499092564 (501h30m26.370175609s) [noise=true , address=145.79.227.45:4231 , piece_id=J3VJJ34DMXNU5U2AYURPXZMTMW5LRMM5JJN4UN3NP2YGZ67J5MOA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.503937500 (501h30m26.375020545s) [noise=true , address=145.79.227.52:4446 , piece_id=KHTOBTC5BYO6XP4D6LWRDDLIH6ZW5XI7JYQAC7Z7GPVCMCDJT3UQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.504793870 (501h30m26.375876905s) [noise=true , address=145.79.227.42:4122 , piece_id=M6VMD2ZL4CBWJD3DMZIG4GJB6JKVI27O3NZCQOFYLB6AFFYWJPYA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.510485258 (501h30m26.381568293s) [noise=true , address=145.79.227.41:4086 , piece_id=SCFWUXVZT7F64FLO6ODQRCKUVJQ4WHT3F7PHBV2HP3JFEO5PC74Q, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.516462209 (501h30m26.387545244s) [noise=true , address=145.79.227.13:4804 , piece_id=3N6LJSS3IF42W6YJGCX2JPSVDKZ63JJHVMQX5KOD2UWJXDOZYTZQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.704044719 (501h30m26.575127754s) [noise=true , address=45.91.116.5:4297 , piece_id=J5A34AC7WW6VPOJLRQNEHHGUARNLUXYRMYPAMFVXVKQITS7CCVQA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.726190408 (501h30m26.597273453s) [noise=true , address=45.91.116.4:4937 , piece_id=T2LAZCFPAQKVVN7UKLGBKFNPPHSMVUNP35QGA3P7BUTHCDV2SHIA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.836341746 (501h30m26.707424791s) [noise=true , address=45.91.116.13:4154 , piece_id=HPDX3QNFWVYCSK5L3J5TCCCYZNKPYDIH6USW2FOUA6QUXGUFXJZQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: <nil> Dec 26 12:59:26.836365040 (501h30m26.707448085s) [seg_pos={0 0}, upload=440131]: Segment reached optimal threshold of 13 pieces. Dec 26 12:59:26.836601220 (501h30m26.707684255s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836617752 (501h30m26.707700787s) [noise=true , address=45.91.116.9:4261 , piece_id=YV74MNR4LGP2EQIMIBAUFDTNNY3CEVYZPLKS3CHHJLYI6L7HDXLA, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836631598 (501h30m26.707714643s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836639072 (501h30m26.707722107s) [noise=true , address=77.90.31.10:4519 , piece_id=F73E5EM5S3NJ6QJLCCZSUOXRC6OQPALRQYNNTQGKERIZCGPLCBCA, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836783136 (501h30m26.707866181s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 12:59:26.836789809 (501h30m26.707872854s) [noise=true , address=145.79.227.43:4158 , piece_id=SVZZFTLRCFJIJZDFEK3QNF4SVP3OPGRVUK22ECN4RQZOJGTBJFHQ, seg_pos={0 0}, upload=440131]: Long tail cancellation due to optimal threshold Dec 26 12:59:26.836805469 (501h30m26.707888504s) [seg_pos={0 0}, upload=440131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:59:26.836909266 (501h30m26.707992301s) [upload=440131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:59:26.836914466 (501h30m26.707997501s) [upload=440131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:59:27.123356883 (501h30m26.994439948s) [upload=440131]: Done uploading segments... Dec 26 12:59:27.123415144 (501h30m26.994498199s) [upload=440131]: Upload finished. err: <nil> Duration: 30.003406493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X....................X.XXX...X..X........] duration: 47.104391249s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:59:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 47.1044073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXXX] duration: 30.002706659s trace: https://jaeger.dev.storj.io/trace/432489002b75732f FAIL (2025-12-26 12:59:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 5MRD2HFNRLVKZBVGCTHD4NJJP52BBN6RHD6TXPDMLNGBOQBJ56AA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: J3BKN6EOTXQIQ536LJOH5H5HTLEGPOU2B4C23B7EMCPUQB5CE6HQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: BYXZMJGAYIBCIIHFIAUEIYDATVMPL6RV36L2JBU43RQ4FFRBTIDA) context deadline exceeded ======LOGS====== ec 26 12:58:50.861720192 (501h29m50.732803227s) [upload=440123]: Starting upload Dec 26 12:58:50.861843777 (501h29m50.732926822s) [upload=440123]: Done starting upload Dec 26 12:58:50.861920182 (501h29m50.733003227s) [upload=440123]: Uploading segments... Dec 26 12:58:50.862073625 (501h29m50.73315666s) [upload=440123]: Got next segment. Inline: false Dec 26 12:58:50.862109583 (501h29m50.733192628s) [upload=440123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:58:50.862118720 (501h29m50.733201755s) [upload=440123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:58:51.108730853 (501h29m50.979813888s) [seg_pos={0 0}, upload=440123]: Begin upload segment... Dec 26 12:58:51.108774266 (501h29m50.979857301s) [seg_pos={0 0}, upload=440123]: Done begin upload segment. Dec 26 12:58:51.108799223 (501h29m50.979882258s) [noise=true , address=45.91.116.6:4974 , piece_id=JSRZRV3OVREKQNMXIT4O6Y5AKLQMYWP7INDRD5X7YKAVFHNNNTLA, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.108882763 (501h29m50.979965798s) [noise=true , address=145.79.227.48:4303 , piece_id=TZTQONPGZNJF7GI2NLXLAY5GIYWDDMY5CYFHHCSCTTR2PYPWUC3A, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.109414906 (501h29m50.980497951s) [noise=true , address=45.91.116.7:4012 , piece_id=UT74E33PW3AECQCDPUD2UWXN3CVKMV4N2H6ARYCT23W2N7Y7WROA, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.109833232 (501h29m50.980916508s) [noise=true , address=45.91.116.12:4226 , piece_id=BYXZMJGAYIBCIIHFIAUEIYDATVMPL6RV36L2JBU43RQ4FFRBTIDA, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.109987547 (501h29m50.981070582s) [noise=true , address=145.79.227.37:4567 , piece_id=KEE4UM45QUCWUHUD6C4GN6LQJUSIKRNCOIAJ6FTA3P7ZRRRCYYFA, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.110496165 (501h29m50.9815792s) [noise=true , address=145.79.227.21:4475 , piece_id=WKILJU73RRFZCGXF7T2A7YCRCYLX6OCZRPYP4FPX6ZIWUXLYVCYQ, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.110736432 (501h29m50.981819477s) [noise=true , address=145.79.227.49:4337 , piece_id=NIPPLBDMMB5H5TUHDC776K5LYNX6YHA3YJ2FBY7CPLLIYIYGSRDQ, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.111127056 (501h29m50.982210091s) [noise=true , address=45.91.116.4:4939 , piece_id=J633X5WHQBTVHFD2LSZA4ZWDY2PZCOZ5MLQTEDK5EGH7DOEEIPJQ, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.111643700 (501h29m50.982726735s) [noise=true , address=45.91.116.9:4261 , piece_id=5NVOAWF4Y7EF2QHHA3YW3KSXMO3J6UDRCZAOV6E2E4A6ZNPLLOIA, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.111892854 (501h29m50.982975889s) [noise=true , address=145.79.227.19:4078 , piece_id=OYVGZ4YGV6R7QMAFRR3O4BFRZPJLKBWY4BFX6ZKYB4OILID7MN3A, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.112335287 (501h29m50.983418322s) [noise=true , address=45.91.116.10:4083 , piece_id=5PVNHELFFXUVVB4CXSJKNRYTIYPBI6VYGCH62UP5WVDD2LOXK5ZQ, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.112685253 (501h29m50.983768288s) [noise=true , address=145.79.227.12:4765 , piece_id=MOKJKICWOJS3FDEHG42KTDL5Z3H5VWT5LYHCZNKQEBQXL2HOW5UQ, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.113140159 (501h29m50.984223194s) [noise=true , address=145.79.227.39:4015 , piece_id=FRVPXT4MODPSYJCR5ZZSS6HUH5P3QKVCTXXWW4D2BB6DH32QP26A, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.113230722 (501h29m50.984313757s) [noise=true , address=145.79.227.6:4259 , piece_id=O4N4T2XYJHF2UP5G3O64AWQG7XLQJ75JNH3WGNN3Q4O6KTPWD5CQ, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.114573619 (501h29m50.985656654s) [noise=true , address=77.90.31.17:4770 , piece_id=5MRD2HFNRLVKZBVGCTHD4NJJP52BBN6RHD6TXPDMLNGBOQBJ56AA, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:51.115092928 (501h29m50.986175963s) [noise=true , address=45.91.116.13:4156 , piece_id=J3BKN6EOTXQIQ536LJOH5H5HTLEGPOU2B4C23B7EMCPUQB5CE6HQ, seg_pos={0 0}, upload=440123]: Uploading piece... Dec 26 12:58:52.821156351 (501h29m52.692239386s) [upload=440123]: Next returned nil segment Dec 26 12:58:52.821169757 (501h29m52.692252792s) [upload=440123]: Waiting for error group managing segments... Dec 26 12:58:53.106940104 (501h29m52.978023139s) [noise=true , address=145.79.227.49:4337 , piece_id=NIPPLBDMMB5H5TUHDC776K5LYNX6YHA3YJ2FBY7CPLLIYIYGSRDQ, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.112266957 (501h29m52.983350003s) [noise=true , address=145.79.227.21:4475 , piece_id=WKILJU73RRFZCGXF7T2A7YCRCYLX6OCZRPYP4FPX6ZIWUXLYVCYQ, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.119639466 (501h29m52.990722511s) [noise=true , address=145.79.227.19:4078 , piece_id=OYVGZ4YGV6R7QMAFRR3O4BFRZPJLKBWY4BFX6ZKYB4OILID7MN3A, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.120469537 (501h29m52.991552572s) [noise=true , address=145.79.227.6:4259 , piece_id=O4N4T2XYJHF2UP5G3O64AWQG7XLQJ75JNH3WGNN3Q4O6KTPWD5CQ, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.122545820 (501h29m52.993628865s) [noise=true , address=145.79.227.12:4765 , piece_id=MOKJKICWOJS3FDEHG42KTDL5Z3H5VWT5LYHCZNKQEBQXL2HOW5UQ, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.136603999 (501h29m53.007687044s) [noise=true , address=145.79.227.48:4303 , piece_id=TZTQONPGZNJF7GI2NLXLAY5GIYWDDMY5CYFHHCSCTTR2PYPWUC3A, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.139184252 (501h29m53.010267297s) [noise=true , address=145.79.227.37:4567 , piece_id=KEE4UM45QUCWUHUD6C4GN6LQJUSIKRNCOIAJ6FTA3P7ZRRRCYYFA, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.144300374 (501h29m53.015383409s) [noise=true , address=145.79.227.39:4015 , piece_id=FRVPXT4MODPSYJCR5ZZSS6HUH5P3QKVCTXXWW4D2BB6DH32QP26A, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.451919696 (501h29m53.323002731s) [noise=true , address=45.91.116.13:4156 , piece_id=J3BKN6EOTXQIQ536LJOH5H5HTLEGPOU2B4C23B7EMCPUQB5CE6HQ, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.466517183 (501h29m53.337600228s) [noise=true , address=45.91.116.12:4226 , piece_id=BYXZMJGAYIBCIIHFIAUEIYDATVMPL6RV36L2JBU43RQ4FFRBTIDA, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:53.487828353 (501h29m53.358911398s) [noise=true , address=45.91.116.6:4974 , piece_id=JSRZRV3OVREKQNMXIT4O6Y5AKLQMYWP7INDRD5X7YKAVFHNNNTLA, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:54.116233516 (501h29m53.987316561s) [noise=true , address=45.91.116.10:4083 , piece_id=5PVNHELFFXUVVB4CXSJKNRYTIYPBI6VYGCH62UP5WVDD2LOXK5ZQ, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:54.204672459 (501h29m54.075755504s) [noise=true , address=77.90.31.17:4770 , piece_id=5MRD2HFNRLVKZBVGCTHD4NJJP52BBN6RHD6TXPDMLNGBOQBJ56AA, seg_pos={0 0}, upload=440123]: Done uploading piece. err: <nil> Dec 26 12:58:54.204698599 (501h29m54.075781634s) [seg_pos={0 0}, upload=440123]: Segment reached optimal threshold of 13 pieces. Dec 26 12:58:54.204917616 (501h29m54.076000651s) [noise=true , address=45.91.116.7:4012 , piece_id=UT74E33PW3AECQCDPUD2UWXN3CVKMV4N2H6ARYCT23W2N7Y7WROA, seg_pos={0 0}, upload=440123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 12:58:54.204926273 (501h29m54.076009308s) [noise=true , address=45.91.116.7:4012 , piece_id=UT74E33PW3AECQCDPUD2UWXN3CVKMV4N2H6ARYCT23W2N7Y7WROA, seg_pos={0 0}, upload=440123]: Long tail cancellation due to optimal threshold Dec 26 12:58:54.205063955 (501h29m54.07614699s) [noise=true , address=45.91.116.9:4261 , piece_id=5NVOAWF4Y7EF2QHHA3YW3KSXMO3J6UDRCZAOV6E2E4A6ZNPLLOIA, seg_pos={0 0}, upload=440123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 12:58:54.205080897 (501h29m54.076163932s) [noise=true , address=45.91.116.9:4261 , piece_id=5NVOAWF4Y7EF2QHHA3YW3KSXMO3J6UDRCZAOV6E2E4A6ZNPLLOIA, seg_pos={0 0}, upload=440123]: Long tail cancellation due to optimal threshold Dec 26 12:58:54.205133778 (501h29m54.076216813s) [noise=true , address=45.91.116.4:4939 , piece_id=J633X5WHQBTVHFD2LSZA4ZWDY2PZCOZ5MLQTEDK5EGH7DOEEIPJQ, seg_pos={0 0}, upload=440123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:53794->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:53794->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53794->45.91.116.4:4939: use of closed network connection Dec 26 12:58:54.205141533 (501h29m54.076224568s) [noise=true , address=45.91.116.4:4939 , piece_id=J633X5WHQBTVHFD2LSZA4ZWDY2PZCOZ5MLQTEDK5EGH7DOEEIPJQ, seg_pos={0 0}, upload=440123]: Long tail cancellation due to optimal threshold Dec 26 12:58:54.205158585 (501h29m54.07624163s) [seg_pos={0 0}, upload=440123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:58:54.205230131 (501h29m54.076313176s) [upload=440123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:58:54.205235652 (501h29m54.076318687s) [upload=440123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:58:54.506622105 (501h29m54.37770514s) [upload=440123]: Done uploading segments... Dec 26 12:58:54.506640019 (501h29m54.377723054s) [upload=440123]: Upload finished. err: <nil> Duration: 30.002719754s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..X..] duration: 5.000869493s trace: https://jaeger.dev.storj.io/trace/4477206e0c24722a FAIL (2025-12-26 12:58:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000886916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X..XXXX] duration: 30.003052277s trace: https://jaeger.dev.storj.io/trace/63d387ef436b4f87 FAIL (2025-12-26 12:58:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: YESGOW4MIDIZQ6QK7RJD43UFOHOTZ2QAVIVL7C2R6KK4MP5NA2KQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 4W5C3FG76O6AFCV4XL6WGZIRRYEHIHLWGRWWEJH2JWPG6FZLTTNQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: OTYFA46C424EL73QLFJP5XBR47Z3NUBXUZB23DXGBXVWBYRJGKDQ) context deadline exceeded ======LOGS====== Dec 26 12:58:20.650046511 (501h29m20.521129557s) [upload=440117]: Starting upload Dec 26 12:58:20.650185656 (501h29m20.521268701s) [upload=440117]: Done starting upload Dec 26 12:58:20.650656453 (501h29m20.521739528s) [upload=440117]: Uploading segments... Dec 26 12:58:20.650712830 (501h29m20.521795865s) [upload=440117]: Got next segment. Inline: false Dec 26 12:58:20.650749810 (501h29m20.521832846s) [upload=440117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:58:20.650755853 (501h29m20.521838898s) [upload=440117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:58:20.897152675 (501h29m20.76823572s) [seg_pos={0 0}, upload=440117]: Begin upload segment... Dec 26 12:58:20.897203412 (501h29m20.768286457s) [seg_pos={0 0}, upload=440117]: Done begin upload segment. Dec 26 12:58:20.897237356 (501h29m20.768320392s) [noise=true , address=45.91.116.14:4190 , piece_id=YESGOW4MIDIZQ6QK7RJD43UFOHOTZ2QAVIVL7C2R6KK4MP5NA2KQ, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.897285638 (501h29m20.768368673s) [noise=true , address=145.79.227.52:4446 , piece_id=4YX2IO5Y2IVEDFOJ3A4H5EOXXNBRPZPNEDGVTO2YPVP3MZLH6KZA, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.897585079 (501h29m20.768668124s) [noise=true , address=45.91.116.9:4261 , piece_id=4W5C3FG76O6AFCV4XL6WGZIRRYEHIHLWGRWWEJH2JWPG6FZLTTNQ, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.898037210 (501h29m20.769120255s) [noise=true , address=145.79.227.48:4302 , piece_id=DU7ZG5VE26DIUFLU6W3TQIPBLK4DBOIPDLNAMRARRKG2TUO46VUQ, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.898485543 (501h29m20.769568588s) [noise=true , address=45.91.116.13:4153 , piece_id=FKP72LME4A3VIW2EIVHR2CKQKJGSMN54ZAS4XBQSVVVDLWKECLHQ, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.898933816 (501h29m20.770016861s) [noise=true , address=145.79.227.35:4483 , piece_id=GIDRX6IMJKGHAJJMF5C54OYBTLG5C3N5KRX5DTMREH46T2WT4HTA, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.899315022 (501h29m20.770398067s) [noise=true , address=145.79.227.43:4159 , piece_id=VPA2PMNPYD263K3A77RARUDIWOMMKNKKPKOLVRJXLYSAEU2SGD4A, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.899703351 (501h29m20.770786396s) [noise=true , address=145.79.227.21:4475 , piece_id=C7ENMTSWXPNJ2M5PBV2PY5BOI73UFGHQR3H4LDSC4ILUS2YKGQUQ, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.900151385 (501h29m20.77123443s) [noise=true , address=77.90.31.11:4556 , piece_id=I7N7RVQ2DGP7EPJHHUUAZGMPDC57AII7BAB5FZVEZK3IQ7HTPFOA, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.901308950 (501h29m20.772391985s) [noise=true , address=77.90.31.19:4844 , piece_id=OTYFA46C424EL73QLFJP5XBR47Z3NUBXUZB23DXGBXVWBYRJGKDQ, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.901742795 (501h29m20.77282584s) [noise=true , address=45.91.116.4:4939 , piece_id=RVNQUGRUWAOT5CV6UV2KVBITFCJ23J6EJ7BCVARZ6CJBLQJWB3XA, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.903255025 (501h29m20.77433807s) [noise=true , address=145.79.227.11:4731 , piece_id=XTNGIR2SQVXTJ4J7EIBOC3ADC7FPKSEEMIHJXUGCEPNKENPYWKQA, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.904697462 (501h29m20.775780507s) [noise=true , address=77.90.31.10:4520 , piece_id=Q3OLOF75D7MRWPRZGR6F4JHJHFE3AZDGANNJEJ23JJGIHUGK24BQ, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.906205322 (501h29m20.777288368s) [noise=true , address=145.79.227.17:4970 , piece_id=GREKBGT6J7PE5NPLWPUIG45AIWOECKGPGVE4ZHWW552EA7263EBA, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.907841869 (501h29m20.778924914s) [noise=true , address=145.79.227.29:4943 , piece_id=CZCCVED7H5RQSNUPLZPIQVL3XMTFRDBPN4PG3I3VZRVHON3YQ7ZA, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:20.909303642 (501h29m20.780386677s) [noise=true , address=45.91.116.6:4973 , piece_id=VF7MAINVOPAQJLDEJY3ZOJUOEXMKROI2P2KKEVQLAAWBDPB4J2CA, seg_pos={0 0}, upload=440117]: Uploading piece... Dec 26 12:58:22.601690394 (501h29m22.472773439s) [upload=440117]: Next returned nil segment Dec 26 12:58:22.601704261 (501h29m22.472787296s) [upload=440117]: Waiting for error group managing segments... Dec 26 12:58:22.889448656 (501h29m22.760531701s) [noise=true , address=145.79.227.52:4446 , piece_id=4YX2IO5Y2IVEDFOJ3A4H5EOXXNBRPZPNEDGVTO2YPVP3MZLH6KZA, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:22.915972783 (501h29m22.787055828s) [noise=true , address=145.79.227.35:4483 , piece_id=GIDRX6IMJKGHAJJMF5C54OYBTLG5C3N5KRX5DTMREH46T2WT4HTA, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:22.918382121 (501h29m22.789465156s) [noise=true , address=145.79.227.48:4302 , piece_id=DU7ZG5VE26DIUFLU6W3TQIPBLK4DBOIPDLNAMRARRKG2TUO46VUQ, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:22.918923642 (501h29m22.790006687s) [noise=true , address=145.79.227.11:4731 , piece_id=XTNGIR2SQVXTJ4J7EIBOC3ADC7FPKSEEMIHJXUGCEPNKENPYWKQA, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:22.919123242 (501h29m22.790206287s) [noise=true , address=145.79.227.29:4943 , piece_id=CZCCVED7H5RQSNUPLZPIQVL3XMTFRDBPN4PG3I3VZRVHON3YQ7ZA, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:22.933079026 (501h29m22.804162061s) [noise=true , address=145.79.227.17:4970 , piece_id=GREKBGT6J7PE5NPLWPUIG45AIWOECKGPGVE4ZHWW552EA7263EBA, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:22.936968693 (501h29m22.808051728s) [noise=true , address=145.79.227.21:4475 , piece_id=C7ENMTSWXPNJ2M5PBV2PY5BOI73UFGHQR3H4LDSC4ILUS2YKGQUQ, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:23.001908519 (501h29m22.872991564s) [noise=true , address=145.79.227.43:4159 , piece_id=VPA2PMNPYD263K3A77RARUDIWOMMKNKKPKOLVRJXLYSAEU2SGD4A, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:23.168923335 (501h29m23.04000638s) [noise=true , address=45.91.116.6:4973 , piece_id=VF7MAINVOPAQJLDEJY3ZOJUOEXMKROI2P2KKEVQLAAWBDPB4J2CA, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:23.173401372 (501h29m23.044484407s) [noise=true , address=45.91.116.13:4153 , piece_id=FKP72LME4A3VIW2EIVHR2CKQKJGSMN54ZAS4XBQSVVVDLWKECLHQ, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:23.254411209 (501h29m23.125494245s) [noise=true , address=45.91.116.9:4261 , piece_id=4W5C3FG76O6AFCV4XL6WGZIRRYEHIHLWGRWWEJH2JWPG6FZLTTNQ, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:23.282956284 (501h29m23.154039319s) [noise=true , address=45.91.116.14:4190 , piece_id=YESGOW4MIDIZQ6QK7RJD43UFOHOTZ2QAVIVL7C2R6KK4MP5NA2KQ, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:23.429647490 (501h29m23.300730535s) [noise=true , address=77.90.31.19:4844 , piece_id=OTYFA46C424EL73QLFJP5XBR47Z3NUBXUZB23DXGBXVWBYRJGKDQ, seg_pos={0 0}, upload=440117]: Done uploading piece. err: <nil> Dec 26 12:58:23.429666576 (501h29m23.300749621s) [seg_pos={0 0}, upload=440117]: Segment reached optimal threshold of 13 pieces. Dec 26 12:58:23.429848893 (501h29m23.300931938s) [noise=true , address=77.90.31.11:4556 , piece_id=I7N7RVQ2DGP7EPJHHUUAZGMPDC57AII7BAB5FZVEZK3IQ7HTPFOA, seg_pos={0 0}, upload=440117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 26 12:58:23.429854724 (501h29m23.300937759s) [noise=true , address=77.90.31.11:4556 , piece_id=I7N7RVQ2DGP7EPJHHUUAZGMPDC57AII7BAB5FZVEZK3IQ7HTPFOA, seg_pos={0 0}, upload=440117]: Long tail cancellation due to optimal threshold Dec 26 12:58:23.429912895 (501h29m23.30099593s) [noise=true , address=45.91.116.4:4939 , piece_id=RVNQUGRUWAOT5CV6UV2KVBITFCJ23J6EJ7BCVARZ6CJBLQJWB3XA, seg_pos={0 0}, upload=440117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:57502->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:57502->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57502->45.91.116.4:4939: use of closed network connection Dec 26 12:58:23.429926771 (501h29m23.301009816s) [noise=true , address=45.91.116.4:4939 , piece_id=RVNQUGRUWAOT5CV6UV2KVBITFCJ23J6EJ7BCVARZ6CJBLQJWB3XA, seg_pos={0 0}, upload=440117]: Long tail cancellation due to optimal threshold Dec 26 12:58:23.430032022 (501h29m23.301115057s) [noise=true , address=77.90.31.10:4520 , piece_id=Q3OLOF75D7MRWPRZGR6F4JHJHFE3AZDGANNJEJ23JJGIHUGK24BQ, seg_pos={0 0}, upload=440117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 12:58:23.430037883 (501h29m23.301120928s) [noise=true , address=77.90.31.10:4520 , piece_id=Q3OLOF75D7MRWPRZGR6F4JHJHFE3AZDGANNJEJ23JJGIHUGK24BQ, seg_pos={0 0}, upload=440117]: Long tail cancellation due to optimal threshold Dec 26 12:58:23.430054565 (501h29m23.30113761s) [seg_pos={0 0}, upload=440117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:58:23.430117365 (501h29m23.3012004s) [upload=440117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:58:23.430122725 (501h29m23.30120577s) [upload=440117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:58:23.716557166 (501h29m23.587640211s) [upload=440117]: Done uploading segments... Dec 26 12:58:23.716580781 (501h29m23.587663827s) [upload=440117]: Upload finished. err: <nil> Duration: 30.003074229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X....................X.XXX...X..X.......] duration: 51.70428895s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:58:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 51.70431568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..X.] duration: 3.685812935s trace: https://jaeger.dev.storj.io/trace/4477206e0c24722a FAIL (2025-12-26 12:58:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.685838483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X..XXX] duration: 30.003301832s trace: https://jaeger.dev.storj.io/trace/686040d0d017fc86 FAIL (2025-12-26 12:57:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 12:57:21.010984431 (501h28m20.882067476s) [upload=440101]: Starting upload Dec 26 12:57:21.011084752 (501h28m20.882167787s) [upload=440101]: Done starting upload Dec 26 12:57:21.011184673 (501h28m20.882267728s) [upload=440101]: Uploading segments... Dec 26 12:57:21.011397879 (501h28m20.882480934s) [upload=440101]: Got next segment. Inline: false Dec 26 12:57:21.011444537 (501h28m20.882527572s) [upload=440101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:57:21.011450118 (501h28m20.882533153s) [upload=440101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:57:21.254748601 (501h28m21.125831646s) [seg_pos={0 0}, upload=440101]: Begin upload segment... Dec 26 12:57:21.254802222 (501h28m21.125885267s) [seg_pos={0 0}, upload=440101]: Done begin upload segment. Dec 26 12:57:21.254842289 (501h28m21.125925324s) [noise=true , address=145.79.227.4:4185 , piece_id=E7ZLZXZB5R6K5QBTMW2HTYJPDQNM3P4JEA44EYILYANBQIIYFTJA, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.254943803 (501h28m21.126026848s) [noise=true , address=145.79.227.46:4601 , piece_id=ASUIE7JB5DAZY6RUTPXC5FTUWR6LYCHCZXCAAOG76GEU5T5KTM6A, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.255364454 (501h28m21.126447499s) [noise=true , address=45.91.116.8:4046 , piece_id=NF7GIAHKA7FEIVA5PJ2PHEXXDRBIYMPSTYLDL2P6NC2TUIFVHSEA, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.255372379 (501h28m21.126455424s) [noise=true , address=145.79.227.3:4149 , piece_id=VX2NU7NXYYMZCKGMBGNLUPD6BAUEKY5MB644IVHFSMVMFE2MZULQ, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.255758394 (501h28m21.126841439s) [noise=true , address=145.79.227.29:4943 , piece_id=UYYANTTCR27UGFBXVHBJCJSGCCTBGO56XG3NJK2PMEY4VLOWT6JQ, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.255770807 (501h28m21.126853842s) [noise=true , address=45.91.116.12:4227 , piece_id=GQBN3KEDT2U7JE7LU7N3CSDPTHIUPFX6FSWACKESUD4A7XY64UIA, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.256053164 (501h28m21.127136199s) [noise=true , address=145.79.227.38:4979 , piece_id=J3YF7SSC7WS42IVVJJ5PSVCJ2XHCRID2AUOD7WDQSNW3KER4LGTQ, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.256173835 (501h28m21.12725687s) [noise=true , address=45.91.116.14:4189 , piece_id=2CCKSNVNUC43GFPASJPZCXVTKDFOQN3TOTQSC7JSUKL26XJXACUA, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.256377493 (501h28m21.127460538s) [noise=true , address=145.79.227.49:4339 , piece_id=YEVBHP2LICXBHLPZOZB445LMRDYUEKFO6FQUTOUAMCU46U23HX4Q, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.256548168 (501h28m21.127631203s) [noise=true , address=77.90.31.10:4520 , piece_id=G642UUHBSODBSMNMK7QCYFBWEUOY333PG526XTKTYVLLSVQUTQPQ, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.256656554 (501h28m21.127739599s) [noise=true , address=145.79.227.42:4123 , piece_id=NPYXYJ3JRVHVNC4BAC2GMFH3WTGO3TRVI76DGGNHVSKPHQANJ3RA, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.256939593 (501h28m21.128022638s) [noise=true , address=45.91.116.6:4973 , piece_id=7ZCIKUMJBPXWQVJJNC66SQIHVPYNKJAFYSIINZEIC5MMSN5QVFLQ, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.256968057 (501h28m21.128051102s) [noise=true , address=45.91.116.5:4300 , piece_id=IQESQZLNIMUPRY23HCB6PXFHBBGYZXCK74XI4A7SPJY2GRGJRWPQ, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.257330236 (501h28m21.128413272s) [noise=true , address=145.79.227.15:4873 , piece_id=BIS7YLZBRQD6CRRVZKXMPW46BJDDIEOB6PHE2QV66ON3IDUEW3IQ, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.257722293 (501h28m21.128805338s) [noise=true , address=145.79.227.40:4049 , piece_id=ARYAVT546OGLVU5IUOSZD32674KIMS5VFCNWITMWKQ3MQ77BFTDQ, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:21.258933700 (501h28m21.130016795s) [noise=true , address=145.79.227.35:4482 , piece_id=F2EF3H6ZW4AOLJBHIMSPISMWDDCLEBKSEYIXOS56SZNJSDZNMD7Q, seg_pos={0 0}, upload=440101]: Uploading piece... Dec 26 12:57:22.960151617 (501h28m22.831234652s) [upload=440101]: Next returned nil segment Dec 26 12:57:22.960176554 (501h28m22.831259599s) [upload=440101]: Waiting for error group managing segments... Dec 26 12:57:23.247940337 (501h28m23.119023372s) [noise=true , address=145.79.227.29:4943 , piece_id=UYYANTTCR27UGFBXVHBJCJSGCCTBGO56XG3NJK2PMEY4VLOWT6JQ, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.257854295 (501h28m23.12893733s) [noise=true , address=145.79.227.40:4049 , piece_id=ARYAVT546OGLVU5IUOSZD32674KIMS5VFCNWITMWKQ3MQ77BFTDQ, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.258991861 (501h28m23.130074897s) [noise=true , address=145.79.227.3:4149 , piece_id=VX2NU7NXYYMZCKGMBGNLUPD6BAUEKY5MB644IVHFSMVMFE2MZULQ, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.261634122 (501h28m23.132717167s) [noise=true , address=145.79.227.49:4339 , piece_id=YEVBHP2LICXBHLPZOZB445LMRDYUEKFO6FQUTOUAMCU46U23HX4Q, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.265276498 (501h28m23.136359533s) [noise=true , address=145.79.227.4:4185 , piece_id=E7ZLZXZB5R6K5QBTMW2HTYJPDQNM3P4JEA44EYILYANBQIIYFTJA, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.267231150 (501h28m23.138314195s) [noise=true , address=145.79.227.46:4601 , piece_id=ASUIE7JB5DAZY6RUTPXC5FTUWR6LYCHCZXCAAOG76GEU5T5KTM6A, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.268980941 (501h28m23.140063977s) [noise=true , address=145.79.227.42:4123 , piece_id=NPYXYJ3JRVHVNC4BAC2GMFH3WTGO3TRVI76DGGNHVSKPHQANJ3RA, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.273616289 (501h28m23.144699334s) [noise=true , address=145.79.227.15:4873 , piece_id=BIS7YLZBRQD6CRRVZKXMPW46BJDDIEOB6PHE2QV66ON3IDUEW3IQ, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.285968480 (501h28m23.157051515s) [noise=true , address=145.79.227.35:4482 , piece_id=F2EF3H6ZW4AOLJBHIMSPISMWDDCLEBKSEYIXOS56SZNJSDZNMD7Q, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.290808035 (501h28m23.16189108s) [noise=true , address=145.79.227.38:4979 , piece_id=J3YF7SSC7WS42IVVJJ5PSVCJ2XHCRID2AUOD7WDQSNW3KER4LGTQ, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.553732099 (501h28m23.424815144s) [noise=true , address=45.91.116.5:4300 , piece_id=IQESQZLNIMUPRY23HCB6PXFHBBGYZXCK74XI4A7SPJY2GRGJRWPQ, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.597579852 (501h28m23.468662887s) [noise=true , address=45.91.116.12:4227 , piece_id=GQBN3KEDT2U7JE7LU7N3CSDPTHIUPFX6FSWACKESUD4A7XY64UIA, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.635729165 (501h28m23.50681221s) [noise=true , address=77.90.31.10:4520 , piece_id=G642UUHBSODBSMNMK7QCYFBWEUOY333PG526XTKTYVLLSVQUTQPQ, seg_pos={0 0}, upload=440101]: Done uploading piece. err: <nil> Dec 26 12:57:23.635771106 (501h28m23.506854141s) [seg_pos={0 0}, upload=440101]: Segment reached optimal threshold of 13 pieces. Dec 26 12:57:23.635979663 (501h28m23.507062708s) [noise=true , address=45.91.116.6:4973 , piece_id=7ZCIKUMJBPXWQVJJNC66SQIHVPYNKJAFYSIINZEIC5MMSN5QVFLQ, seg_pos={0 0}, upload=440101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 12:57:23.635986045 (501h28m23.50706909s) [noise=true , address=45.91.116.6:4973 , piece_id=7ZCIKUMJBPXWQVJJNC66SQIHVPYNKJAFYSIINZEIC5MMSN5QVFLQ, seg_pos={0 0}, upload=440101]: Long tail cancellation due to optimal threshold Dec 26 12:57:23.636232814 (501h28m23.50731586s) [noise=true , address=45.91.116.14:4189 , piece_id=2CCKSNVNUC43GFPASJPZCXVTKDFOQN3TOTQSC7JSUKL26XJXACUA, seg_pos={0 0}, upload=440101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 12:57:23.636248193 (501h28m23.507331228s) [noise=true , address=45.91.116.14:4189 , piece_id=2CCKSNVNUC43GFPASJPZCXVTKDFOQN3TOTQSC7JSUKL26XJXACUA, seg_pos={0 0}, upload=440101]: Long tail cancellation due to optimal threshold Dec 26 12:57:23.636446141 (501h28m23.507529186s) [noise=true , address=45.91.116.8:4046 , piece_id=NF7GIAHKA7FEIVA5PJ2PHEXXDRBIYMPSTYLDL2P6NC2TUIFVHSEA, seg_pos={0 0}, upload=440101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:38072->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:57:23.636454105 (501h28m23.507537151s) [noise=true , address=45.91.116.8:4046 , piece_id=NF7GIAHKA7FEIVA5PJ2PHEXXDRBIYMPSTYLDL2P6NC2TUIFVHSEA, seg_pos={0 0}, upload=440101]: Long tail cancellation due to optimal threshold Dec 26 12:57:23.636477270 (501h28m23.507560305s) [seg_pos={0 0}, upload=440101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:57:23.636566960 (501h28m23.507650025s) [upload=440101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:57:23.636575978 (501h28m23.507659013s) [upload=440101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:57:23.923484972 (501h28m23.794568007s) [upload=440101]: Done uploading segments... Dec 26 12:57:23.923504580 (501h28m23.794587615s) [upload=440101]: Upload finished. err: <nil> Duration: 30.003325306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..X] duration: 5.001941776s trace: https://jaeger.dev.storj.io/trace/4477206e0c24722a FAIL (2025-12-26 12:57:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001965532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X....................X.XXX...X..X......] duration: 50.784601416s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:57:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 50.784617796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X..XX] duration: 30.002424461s trace: https://jaeger.dev.storj.io/trace/2bacd0b5c41314aa FAIL (2025-12-26 12:56:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 35MVXXRAQJUB7KJTJXSSJMXT5YTGSRDBNT53GA56BISC5VQDSH3A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: IVZYUWKLUGKWRK7G6XPEZBPQZNPZYS4NKW7KY5KCOOGJCB4DCXZQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: T3EIUSETNYTI2VVX6NTJ434LKK2QB6IHK7AVYYNH67C46XFWTI5Q) context deadline exceeded ======LOGS====== 239211 (501h27m24.445322246s) [upload=440089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:56:24.818737418 (501h27m24.689820473s) [seg_pos={0 0}, upload=440089]: Begin upload segment... Dec 26 12:56:24.818815887 (501h27m24.689898932s) [seg_pos={0 0}, upload=440089]: Done begin upload segment. Dec 26 12:56:24.818869630 (501h27m24.689952675s) [noise=true , address=145.79.227.35:4481 , piece_id=MGMTYZYBUS3NVHJIHVHCSDOSHXPJEDGTHNO56KWS4W3TWS5B66XQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.818883837 (501h27m24.689966872s) [noise=true , address=145.79.227.13:4804 , piece_id=UDDITPL6EXWAB6LHWTB3DPI6FQVHAHDROOYHR7DX3YGNXEWWDCHQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.819063398 (501h27m24.690146433s) [noise=true , address=77.90.31.19:4844 , piece_id=4I5SFQ3PKIEP2VUTMWNEAVHQVDP7AEZFOI23OKBFIZDHL47VQLWA, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.819673881 (501h27m24.690756916s) [noise=true , address=45.91.116.10:4083 , piece_id=DE3CDAMDUDIOJOLGN7JIWUHJJROQUBGVW6AHBJ2AT3Z5BNG6H4CQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.820078683 (501h27m24.691161718s) [noise=true , address=45.91.116.13:4156 , piece_id=T3EIUSETNYTI2VVX6NTJ434LKK2QB6IHK7AVYYNH67C46XFWTI5Q, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.820183312 (501h27m24.691266357s) [noise=true , address=77.90.31.13:4629 , piece_id=4GG7UYUCDL6DVVRGM37EIY7IDJYUYF376OTUKWQISJWA52BAJ4CQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.820632707 (501h27m24.691715742s) [noise=true , address=45.91.116.7:4009 , piece_id=KBA3WZA75EVYGCO42XSIVH6EDRPKCTLP274YOMKKYTXKZRD3EMAQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.821001129 (501h27m24.692084164s) [noise=true , address=45.91.116.8:4047 , piece_id=35MVXXRAQJUB7KJTJXSSJMXT5YTGSRDBNT53GA56BISC5VQDSH3A, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.821370923 (501h27m24.692453958s) [noise=true , address=45.91.116.14:4190 , piece_id=IVZYUWKLUGKWRK7G6XPEZBPQZNPZYS4NKW7KY5KCOOGJCB4DCXZQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.821787356 (501h27m24.692870391s) [noise=true , address=145.79.227.11:4730 , piece_id=2WLRJKAC2DFY4P63YMZNE5IWKUSZ2GRDJNONJXPZ5JZMAHY4D5MQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.822010200 (501h27m24.693093235s) [noise=true , address=145.79.227.48:4303 , piece_id=DKRS2QLCRAOQLVA6FUSKJA7EWJ4YFQ4DSKIGHE7USSBX6TIDFEMQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.822781278 (501h27m24.693864323s) [noise=true , address=145.79.227.42:4121 , piece_id=N3W3ESXDENUR4H5XTDWBIOAJ6GO7ZAHKLZ6THP4RJCYBKSYZSRBA, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.823230443 (501h27m24.694313479s) [noise=true , address=77.90.31.14:4664 , piece_id=CS7RVDOESKPHCBL7JTFGZ53PK2TUUE6RFA5QX5465HCPLIFHDZOA, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.824127181 (501h27m24.695210216s) [noise=true , address=145.79.227.49:4339 , piece_id=IEIFYG7FH3ZMF4DYL3NK5LHQLQRUVSSUZ2OMPSWIGC6MSW5N7EYA, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.824491304 (501h27m24.695574339s) [noise=true , address=45.91.116.11:4119 , piece_id=2TKRBXRUITQDWTHI7NS52UV2C4VBNP2OP2TH5B2N72P3PJ6GFXWQ, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:24.824834478 (501h27m24.695917513s) [noise=true , address=145.79.227.39:4014 , piece_id=RIOBCZOYMWCUOI3YASGEEP3HIG3XWVFCXI2DCTD523MTD5ZPFL3Q, seg_pos={0 0}, upload=440089]: Uploading piece... Dec 26 12:56:26.539974516 (501h27m26.411057561s) [upload=440089]: Next returned nil segment Dec 26 12:56:26.539987480 (501h27m26.411070515s) [upload=440089]: Waiting for error group managing segments... Dec 26 12:56:26.818053247 (501h27m26.689136282s) [noise=true , address=145.79.227.39:4014 , piece_id=RIOBCZOYMWCUOI3YASGEEP3HIG3XWVFCXI2DCTD523MTD5ZPFL3Q, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:26.831882720 (501h27m26.702965755s) [noise=true , address=145.79.227.42:4121 , piece_id=N3W3ESXDENUR4H5XTDWBIOAJ6GO7ZAHKLZ6THP4RJCYBKSYZSRBA, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:26.838363721 (501h27m26.709446756s) [noise=true , address=145.79.227.13:4804 , piece_id=UDDITPL6EXWAB6LHWTB3DPI6FQVHAHDROOYHR7DX3YGNXEWWDCHQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:26.847027378 (501h27m26.718110413s) [noise=true , address=145.79.227.11:4730 , piece_id=2WLRJKAC2DFY4P63YMZNE5IWKUSZ2GRDJNONJXPZ5JZMAHY4D5MQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:26.848753195 (501h27m26.71983623s) [noise=true , address=145.79.227.48:4303 , piece_id=DKRS2QLCRAOQLVA6FUSKJA7EWJ4YFQ4DSKIGHE7USSBX6TIDFEMQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:26.849527719 (501h27m26.720610764s) [noise=true , address=145.79.227.35:4481 , piece_id=MGMTYZYBUS3NVHJIHVHCSDOSHXPJEDGTHNO56KWS4W3TWS5B66XQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:26.852933044 (501h27m26.724016079s) [noise=true , address=145.79.227.49:4339 , piece_id=IEIFYG7FH3ZMF4DYL3NK5LHQLQRUVSSUZ2OMPSWIGC6MSW5N7EYA, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:27.156078997 (501h27m27.027162032s) [noise=true , address=45.91.116.8:4047 , piece_id=35MVXXRAQJUB7KJTJXSSJMXT5YTGSRDBNT53GA56BISC5VQDSH3A, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:27.185632772 (501h27m27.056715817s) [noise=true , address=45.91.116.7:4009 , piece_id=KBA3WZA75EVYGCO42XSIVH6EDRPKCTLP274YOMKKYTXKZRD3EMAQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:27.188967221 (501h27m27.060050256s) [noise=true , address=45.91.116.10:4083 , piece_id=DE3CDAMDUDIOJOLGN7JIWUHJJROQUBGVW6AHBJ2AT3Z5BNG6H4CQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:27.198978023 (501h27m27.070061059s) [noise=true , address=45.91.116.14:4190 , piece_id=IVZYUWKLUGKWRK7G6XPEZBPQZNPZYS4NKW7KY5KCOOGJCB4DCXZQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:27.212222714 (501h27m27.083305759s) [noise=true , address=77.90.31.13:4629 , piece_id=4GG7UYUCDL6DVVRGM37EIY7IDJYUYF376OTUKWQISJWA52BAJ4CQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:27.313897190 (501h27m27.184980225s) [noise=true , address=45.91.116.13:4156 , piece_id=T3EIUSETNYTI2VVX6NTJ434LKK2QB6IHK7AVYYNH67C46XFWTI5Q, seg_pos={0 0}, upload=440089]: Done uploading piece. err: <nil> Dec 26 12:56:27.313929250 (501h27m27.185012295s) [seg_pos={0 0}, upload=440089]: Segment reached optimal threshold of 13 pieces. Dec 26 12:56:27.314169197 (501h27m27.185252242s) [noise=true , address=45.91.116.11:4119 , piece_id=2TKRBXRUITQDWTHI7NS52UV2C4VBNP2OP2TH5B2N72P3PJ6GFXWQ, seg_pos={0 0}, upload=440089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:49844->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:49844->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49844->45.91.116.11:4119: use of closed network connection Dec 26 12:56:27.314178666 (501h27m27.185261711s) [noise=true , address=45.91.116.11:4119 , piece_id=2TKRBXRUITQDWTHI7NS52UV2C4VBNP2OP2TH5B2N72P3PJ6GFXWQ, seg_pos={0 0}, upload=440089]: Long tail cancellation due to optimal threshold Dec 26 12:56:27.314529574 (501h27m27.185612619s) [noise=true , address=77.90.31.14:4664 , piece_id=CS7RVDOESKPHCBL7JTFGZ53PK2TUUE6RFA5QX5465HCPLIFHDZOA, seg_pos={0 0}, upload=440089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:38574->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:38574->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38574->77.90.31.14:4664: use of closed network connection Dec 26 12:56:27.314541166 (501h27m27.185624201s) [noise=true , address=77.90.31.14:4664 , piece_id=CS7RVDOESKPHCBL7JTFGZ53PK2TUUE6RFA5QX5465HCPLIFHDZOA, seg_pos={0 0}, upload=440089]: Long tail cancellation due to optimal threshold Dec 26 12:56:27.314692143 (501h27m27.185775188s) [noise=true , address=77.90.31.19:4844 , piece_id=4I5SFQ3PKIEP2VUTMWNEAVHQVDP7AEZFOI23OKBFIZDHL47VQLWA, seg_pos={0 0}, upload=440089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:40014->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:40014->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40014->77.90.31.19:4844: use of closed network connection Dec 26 12:56:27.314700490 (501h27m27.185783535s) [noise=true , address=77.90.31.19:4844 , piece_id=4I5SFQ3PKIEP2VUTMWNEAVHQVDP7AEZFOI23OKBFIZDHL47VQLWA, seg_pos={0 0}, upload=440089]: Long tail cancellation due to optimal threshold Dec 26 12:56:27.314722461 (501h27m27.185805496s) [seg_pos={0 0}, upload=440089]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:56:27.314806061 (501h27m27.185889106s) [upload=440089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:56:27.314813074 (501h27m27.185896119s) [upload=440089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:56:27.607606464 (501h27m27.478689509s) [upload=440089]: Done uploading segments... Dec 26 12:56:27.607630390 (501h27m27.478713425s) [upload=440089]: Upload finished. err: <nil> Duration:* 30.002452457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X..] duration: 3.783032147s trace: https://jaeger.dev.storj.io/trace/3060f5237e3dbb77 FAIL (2025-12-26 12:56:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.783076452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X....................X.XXX...X..X.....] duration: 57.386753819s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:56:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 57.386783637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X..X] duration: 30.002144958s trace: https://jaeger.dev.storj.io/trace/7bc071fc5ed8f989 FAIL (2025-12-26 12:56:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 3B3MRTZHTOC4CZSFY3VEYHRUYOBAZ2LUKIIHQCV5LRBNIP6AOGUA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RMDFGLVEFSU3UKFYGG6HRBQ5FZEHJJGVBERDLLWLWRYGBHZ2BAUQ) context deadline exceeded ======LOGS====== 6 12:55:35.882560210 (501h26m35.753643275s) [upload=440078]: Starting upload Dec 26 12:55:35.882649621 (501h26m35.753732686s) [upload=440078]: Done starting upload Dec 26 12:55:35.882714984 (501h26m35.753798029s) [upload=440078]: Uploading segments... Dec 26 12:55:35.882987513 (501h26m35.754070548s) [upload=440078]: Got next segment. Inline: false Dec 26 12:55:35.883016970 (501h26m35.754100005s) [upload=440078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:55:35.883021929 (501h26m35.754104974s) [upload=440078]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:55:36.130587909 (501h26m36.001670954s) [seg_pos={0 0}, upload=440078]: Begin upload segment... Dec 26 12:55:36.130661308 (501h26m36.001744353s) [seg_pos={0 0}, upload=440078]: Done begin upload segment. Dec 26 12:55:36.130714529 (501h26m36.001797564s) [noise=true , address=145.79.227.2:4113 , piece_id=A6PVWRNEEU7KLBCEFWFC4TFWRA35Z7OKG6DNYHDGBMS4AJEHOYDQ, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.130751830 (501h26m36.001834875s) [noise=true , address=45.91.116.12:4228 , piece_id=BEKDFZ37GB4R3AGCFRESLRLF35Y77GGD67BQW727YN4VW2UAJD3Q, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.131324200 (501h26m36.002407235s) [noise=true , address=145.79.227.11:4730 , piece_id=EZPVOJ5BMN323GPXD36BFF3RL7OF5VOTDZ52KJEIG67DBMMYX7VA, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.131436664 (501h26m36.002519709s) [noise=true , address=145.79.227.4:4185 , piece_id=QWSRGU4OMVYYKWWWQTTKGK4ZAFJC745C4QOQ2UBSKC3OKF6O2DTQ, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.131672504 (501h26m36.002755539s) [noise=true , address=45.91.116.6:4976 , piece_id=7KFQYQ6BJMPVOJRDNJGGXAIP26LTSZGQANOETMKEE4K7SB462OBQ, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.131838028 (501h26m36.002921063s) [noise=true , address=77.90.31.10:4517 , piece_id=3B3MRTZHTOC4CZSFY3VEYHRUYOBAZ2LUKIIHQCV5LRBNIP6AOGUA, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.132000097 (501h26m36.003083142s) [noise=true , address=145.79.227.13:4802 , piece_id=NDKUWKZT5YHB2FLNXQ2O5VITOISHGOQN3GKCZF5UCC6A5FCCVJKA, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.132253890 (501h26m36.003336935s) [noise=true , address=77.90.31.14:4663 , piece_id=RMDFGLVEFSU3UKFYGG6HRBQ5FZEHJJGVBERDLLWLWRYGBHZ2BAUQ, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.132310568 (501h26m36.003393603s) [noise=true , address=145.79.227.15:4874 , piece_id=CKOOQEDV2LEYDKAZX5YNW6Z4QF4QCGPH5DI3J4CYC4DOKAJI6SHQ, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.132664973 (501h26m36.003748018s) [noise=true , address=145.79.227.39:4015 , piece_id=ETMWLSS5DV5X2F7SD2Z4C3D5HFJ4J4J4GWH2QIE6DUB2KRISIRAQ, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.132683529 (501h26m36.003766574s) [noise=true , address=145.79.227.17:4970 , piece_id=A5Y6VOJA7NZ6SYNRLGPDEZWOVNJNC5CF4TKPJJIGOGHMLEW5OITA, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.133039236 (501h26m36.004122281s) [noise=true , address=145.79.227.37:4566 , piece_id=SCYZ5G5XJ3DGUTECKBFRPMINE5AZ4MZIL7AGYAKPQHCDIUNYSJ4A, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.133092487 (501h26m36.004175532s) [noise=true , address=145.79.227.31:4799 , piece_id=XTZRLP3UHBQYEQP4A23CI7ABVCAW3TTQ3E4YQBLVEEBHISEFGKUA, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.133367751 (501h26m36.004450796s) [noise=true , address=145.79.227.21:4475 , piece_id=SVIYPTRDY2ZEP2SNNCGPEXU5XK7GRBWR6N5XHD25BSWGYKLC53XQ, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.133552574 (501h26m36.004635609s) [noise=true , address=145.79.227.49:4338 , piece_id=A5N2NRMKJRBVGD4QM2APR3I7KVQDTG5IPFEYE3Q4PMBKG6QEUX4A, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:36.134049610 (501h26m36.005132655s) [noise=true , address=45.91.116.11:4119 , piece_id=RNJAXRRQL7JB6NKLSGGOWH7B5JVLQZK6NWSFYILKRG4HNNTAEAHA, seg_pos={0 0}, upload=440078]: Uploading piece... Dec 26 12:55:37.851470090 (501h26m37.722553125s) [upload=440078]: Next returned nil segment Dec 26 12:55:37.851482894 (501h26m37.722565929s) [upload=440078]: Waiting for error group managing segments... Dec 26 12:55:38.139344935 (501h26m38.01042797s) [noise=true , address=145.79.227.4:4185 , piece_id=QWSRGU4OMVYYKWWWQTTKGK4ZAFJC745C4QOQ2UBSKC3OKF6O2DTQ, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.139644384 (501h26m38.010727439s) [noise=true , address=145.79.227.17:4970 , piece_id=A5Y6VOJA7NZ6SYNRLGPDEZWOVNJNC5CF4TKPJJIGOGHMLEW5OITA, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.140553956 (501h26m38.011637001s) [noise=true , address=145.79.227.31:4799 , piece_id=XTZRLP3UHBQYEQP4A23CI7ABVCAW3TTQ3E4YQBLVEEBHISEFGKUA, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.140762844 (501h26m38.011845889s) [noise=true , address=145.79.227.37:4566 , piece_id=SCYZ5G5XJ3DGUTECKBFRPMINE5AZ4MZIL7AGYAKPQHCDIUNYSJ4A, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.143666063 (501h26m38.014749108s) [noise=true , address=145.79.227.11:4730 , piece_id=EZPVOJ5BMN323GPXD36BFF3RL7OF5VOTDZ52KJEIG67DBMMYX7VA, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.143706339 (501h26m38.014789384s) [noise=true , address=145.79.227.2:4113 , piece_id=A6PVWRNEEU7KLBCEFWFC4TFWRA35Z7OKG6DNYHDGBMS4AJEHOYDQ, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.144429697 (501h26m38.015512742s) [noise=true , address=145.79.227.13:4802 , piece_id=NDKUWKZT5YHB2FLNXQ2O5VITOISHGOQN3GKCZF5UCC6A5FCCVJKA, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.157440742 (501h26m38.028523777s) [noise=true , address=145.79.227.15:4874 , piece_id=CKOOQEDV2LEYDKAZX5YNW6Z4QF4QCGPH5DI3J4CYC4DOKAJI6SHQ, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.159272109 (501h26m38.030355144s) [noise=true , address=145.79.227.39:4015 , piece_id=ETMWLSS5DV5X2F7SD2Z4C3D5HFJ4J4J4GWH2QIE6DUB2KRISIRAQ, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.163653622 (501h26m38.034736657s) [noise=true , address=145.79.227.21:4475 , piece_id=SVIYPTRDY2ZEP2SNNCGPEXU5XK7GRBWR6N5XHD25BSWGYKLC53XQ, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.164962515 (501h26m38.03604556s) [noise=true , address=145.79.227.49:4338 , piece_id=A5N2NRMKJRBVGD4QM2APR3I7KVQDTG5IPFEYE3Q4PMBKG6QEUX4A, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.407605178 (501h26m38.278688223s) [noise=true , address=77.90.31.10:4517 , piece_id=3B3MRTZHTOC4CZSFY3VEYHRUYOBAZ2LUKIIHQCV5LRBNIP6AOGUA, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.443386061 (501h26m38.314469106s) [noise=true , address=77.90.31.14:4663 , piece_id=RMDFGLVEFSU3UKFYGG6HRBQ5FZEHJJGVBERDLLWLWRYGBHZ2BAUQ, seg_pos={0 0}, upload=440078]: Done uploading piece. err: <nil> Dec 26 12:55:38.443409236 (501h26m38.314492281s) [seg_pos={0 0}, upload=440078]: Segment reached optimal threshold of 13 pieces. Dec 26 12:55:38.443623554 (501h26m38.314706589s) [noise=true , address=45.91.116.6:4976 , piece_id=7KFQYQ6BJMPVOJRDNJGGXAIP26LTSZGQANOETMKEE4K7SB462OBQ, seg_pos={0 0}, upload=440078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:49506->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:49506->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49506->45.91.116.6:4976: use of closed network connection Dec 26 12:55:38.443632140 (501h26m38.314715175s) [noise=true , address=45.91.116.6:4976 , piece_id=7KFQYQ6BJMPVOJRDNJGGXAIP26LTSZGQANOETMKEE4K7SB462OBQ, seg_pos={0 0}, upload=440078]: Long tail cancellation due to optimal threshold Dec 26 12:55:38.443923655 (501h26m38.315006701s) [noise=true , address=45.91.116.11:4119 , piece_id=RNJAXRRQL7JB6NKLSGGOWH7B5JVLQZK6NWSFYILKRG4HNNTAEAHA, seg_pos={0 0}, upload=440078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 12:55:38.443932061 (501h26m38.315015106s) [noise=true , address=45.91.116.11:4119 , piece_id=RNJAXRRQL7JB6NKLSGGOWH7B5JVLQZK6NWSFYILKRG4HNNTAEAHA, seg_pos={0 0}, upload=440078]: Long tail cancellation due to optimal threshold Dec 26 12:55:38.443999860 (501h26m38.315082905s) [noise=true , address=45.91.116.12:4228 , piece_id=BEKDFZ37GB4R3AGCFRESLRLF35Y77GGD67BQW727YN4VW2UAJD3Q, seg_pos={0 0}, upload=440078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 12:55:38.444020590 (501h26m38.315103635s) [noise=true , address=45.91.116.12:4228 , piece_id=BEKDFZ37GB4R3AGCFRESLRLF35Y77GGD67BQW727YN4VW2UAJD3Q, seg_pos={0 0}, upload=440078]: Long tail cancellation due to optimal threshold Dec 26 12:55:38.444051880 (501h26m38.315134925s) [seg_pos={0 0}, upload=440078]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:55:38.444209980 (501h26m38.315293016s) [upload=440078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:55:38.444217915 (501h26m38.31530096s) [upload=440078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:55:38.749118460 (501h26m38.620201505s) [upload=440078]: Done uploading segments... Dec 26 12:55:38.749137947 (501h26m38.620220982s) [upload=440078]: Upload finished. err: <nil> Duration: 30.002168203s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X.] duration: 4.6496242s trace: https://jaeger.dev.storj.io/trace/3060f5237e3dbb77 FAIL (2025-12-26 12:55:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.649650281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................X....................X.XXX...X..X....] duration: 37.822172884s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:55:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 37.822198792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X..] duration: 5.98706514s trace: https://jaeger.dev.storj.io/trace/25f0a86b327adac5 FAIL (2025-12-26 12:54:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ) context deadline exceeded ======LOGS====== segments... Dec 26 12:53:33.060139405 (501h24m32.93122247s) [upload=440045]: Got next segment. Inline: false Dec 26 12:53:33.060183419 (501h24m32.931266454s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:53:33.060189249 (501h24m32.931272295s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:53:33.304580344 (501h24m33.175663389s) [seg_pos={0 0}, upload=440045]: Begin upload segment... Dec 26 12:53:33.304632342 (501h24m33.175715377s) [seg_pos={0 0}, upload=440045]: Done begin upload segment. Dec 26 12:53:33.304685023 (501h24m33.175768068s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304695913 (501h24m33.175778959s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304862942 (501h24m33.175945977s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305344799 (501h24m33.176427844s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305502649 (501h24m33.176585685s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305928852 (501h24m33.177011887s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310305796 (501h24m33.181388851s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310942769 (501h24m33.182025814s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311429736 (501h24m33.182512781s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311996105 (501h24m33.18307915s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.312483022 (501h24m33.183566067s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.313996513 (501h24m33.185079558s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.315528761 (501h24m33.186611806s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.316907637 (501h24m33.187990682s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.319905165 (501h24m33.19098822s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.321664966 (501h24m33.192748001s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:34.914820462 (501h24m34.785903507s) [upload=440045]: Next returned nil segment Dec 26 12:53:34.914837635 (501h24m34.78592069s) [upload=440045]: Waiting for error group managing segments... Dec 26 12:53:35.298879578 (501h24m35.169962623s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.308194576 (501h24m35.179277611s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.321282087 (501h24m35.192365122s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.324654759 (501h24m35.195737804s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.328057719 (501h24m35.199140754s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336066038 (501h24m35.207149073s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336911889 (501h24m35.207994934s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344439162 (501h24m35.215522227s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344909568 (501h24m35.215992603s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.351561845 (501h24m35.22264488s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.352409338 (501h24m35.223492383s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.601492897 (501h24m35.472575942s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610741338 (501h24m35.481824383s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610765683 (501h24m35.481848729s) [seg_pos={0 0}, upload=440045]: Segment reached optimal threshold of 13 pieces. Dec 26 12:53:35.610962629 (501h24m35.482045674s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:53:35.610968921 (501h24m35.482051956s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611078359 (501h24m35.482161394s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection Dec 26 12:53:35.611088449 (501h24m35.482171494s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611146599 (501h24m35.482229634s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection Dec 26 12:53:35.611152770 (501h24m35.482235816s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611193359 (501h24m35.482276404s) [seg_pos={0 0}, upload=440045]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:53:35.611282528 (501h24m35.482365583s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:53:35.611289651 (501h24m35.482372686s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:53:35.896282848 (501h24m35.767365894s) [upload=440045]: Done uploading segments... Dec 26 12:53:35.896302896 (501h24m35.767385931s) [upload=440045]: Upload finished. err: <nil> Duration: 5.987091531s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X] duration: 5.001458211s trace: https://jaeger.dev.storj.io/trace/3060f5237e3dbb77 FAIL (2025-12-26 12:54:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001480875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X.] duration: 5.898239159s trace: https://jaeger.dev.storj.io/trace/25f0a86b327adac5 FAIL (2025-12-26 12:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ) context deadline exceeded ======LOGS====== segments... Dec 26 12:53:33.060139405 (501h24m32.93122247s) [upload=440045]: Got next segment. Inline: false Dec 26 12:53:33.060183419 (501h24m32.931266454s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:53:33.060189249 (501h24m32.931272295s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:53:33.304580344 (501h24m33.175663389s) [seg_pos={0 0}, upload=440045]: Begin upload segment... Dec 26 12:53:33.304632342 (501h24m33.175715377s) [seg_pos={0 0}, upload=440045]: Done begin upload segment. Dec 26 12:53:33.304685023 (501h24m33.175768068s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304695913 (501h24m33.175778959s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304862942 (501h24m33.175945977s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305344799 (501h24m33.176427844s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305502649 (501h24m33.176585685s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305928852 (501h24m33.177011887s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310305796 (501h24m33.181388851s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310942769 (501h24m33.182025814s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311429736 (501h24m33.182512781s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311996105 (501h24m33.18307915s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.312483022 (501h24m33.183566067s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.313996513 (501h24m33.185079558s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.315528761 (501h24m33.186611806s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.316907637 (501h24m33.187990682s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.319905165 (501h24m33.19098822s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.321664966 (501h24m33.192748001s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:34.914820462 (501h24m34.785903507s) [upload=440045]: Next returned nil segment Dec 26 12:53:34.914837635 (501h24m34.78592069s) [upload=440045]: Waiting for error group managing segments... Dec 26 12:53:35.298879578 (501h24m35.169962623s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.308194576 (501h24m35.179277611s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.321282087 (501h24m35.192365122s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.324654759 (501h24m35.195737804s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.328057719 (501h24m35.199140754s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336066038 (501h24m35.207149073s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336911889 (501h24m35.207994934s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344439162 (501h24m35.215522227s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344909568 (501h24m35.215992603s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.351561845 (501h24m35.22264488s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.352409338 (501h24m35.223492383s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.601492897 (501h24m35.472575942s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610741338 (501h24m35.481824383s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610765683 (501h24m35.481848729s) [seg_pos={0 0}, upload=440045]: Segment reached optimal threshold of 13 pieces. Dec 26 12:53:35.610962629 (501h24m35.482045674s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:53:35.610968921 (501h24m35.482051956s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611078359 (501h24m35.482161394s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection Dec 26 12:53:35.611088449 (501h24m35.482171494s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611146599 (501h24m35.482229634s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection Dec 26 12:53:35.611152770 (501h24m35.482235816s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611193359 (501h24m35.482276404s) [seg_pos={0 0}, upload=440045]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:53:35.611282528 (501h24m35.482365583s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:53:35.611289651 (501h24m35.482372686s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:53:35.896282848 (501h24m35.767365894s) [upload=440045]: Done uploading segments... Dec 26 12:53:35.896302896 (501h24m35.767385931s) [upload=440045]: Upload finished. err: <nil> Duration: 5.898272643s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X....................X.XXX...X..X...] duration: 39.659163378s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:54:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 39.659191693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X] duration: 30.004943123s trace: https://jaeger.dev.storj.io/trace/25f0a86b327adac5 FAIL (2025-12-26 12:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ) context deadline exceeded ======LOGS====== segments... Dec 26 12:53:33.060139405 (501h24m32.93122247s) [upload=440045]: Got next segment. Inline: false Dec 26 12:53:33.060183419 (501h24m32.931266454s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:53:33.060189249 (501h24m32.931272295s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:53:33.304580344 (501h24m33.175663389s) [seg_pos={0 0}, upload=440045]: Begin upload segment... Dec 26 12:53:33.304632342 (501h24m33.175715377s) [seg_pos={0 0}, upload=440045]: Done begin upload segment. Dec 26 12:53:33.304685023 (501h24m33.175768068s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304695913 (501h24m33.175778959s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304862942 (501h24m33.175945977s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305344799 (501h24m33.176427844s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305502649 (501h24m33.176585685s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305928852 (501h24m33.177011887s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310305796 (501h24m33.181388851s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310942769 (501h24m33.182025814s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311429736 (501h24m33.182512781s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311996105 (501h24m33.18307915s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.312483022 (501h24m33.183566067s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.313996513 (501h24m33.185079558s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.315528761 (501h24m33.186611806s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.316907637 (501h24m33.187990682s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.319905165 (501h24m33.19098822s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.321664966 (501h24m33.192748001s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:34.914820462 (501h24m34.785903507s) [upload=440045]: Next returned nil segment Dec 26 12:53:34.914837635 (501h24m34.78592069s) [upload=440045]: Waiting for error group managing segments... Dec 26 12:53:35.298879578 (501h24m35.169962623s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.308194576 (501h24m35.179277611s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.321282087 (501h24m35.192365122s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.324654759 (501h24m35.195737804s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.328057719 (501h24m35.199140754s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336066038 (501h24m35.207149073s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336911889 (501h24m35.207994934s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344439162 (501h24m35.215522227s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344909568 (501h24m35.215992603s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.351561845 (501h24m35.22264488s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.352409338 (501h24m35.223492383s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.601492897 (501h24m35.472575942s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610741338 (501h24m35.481824383s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610765683 (501h24m35.481848729s) [seg_pos={0 0}, upload=440045]: Segment reached optimal threshold of 13 pieces. Dec 26 12:53:35.610962629 (501h24m35.482045674s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:53:35.610968921 (501h24m35.482051956s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611078359 (501h24m35.482161394s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection Dec 26 12:53:35.611088449 (501h24m35.482171494s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611146599 (501h24m35.482229634s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection Dec 26 12:53:35.611152770 (501h24m35.482235816s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611193359 (501h24m35.482276404s) [seg_pos={0 0}, upload=440045]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:53:35.611282528 (501h24m35.482365583s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:53:35.611289651 (501h24m35.482372686s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:53:35.896282848 (501h24m35.767365894s) [upload=440045]: Done uploading segments... Dec 26 12:53:35.896302896 (501h24m35.767385931s) [upload=440045]: Upload finished. err: <nil> Duration: 30.004969061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X....................X.XXX...X..X..] duration: 44.881595442s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:53:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 44.881624407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.] duration: 3.531158312s trace: https://jaeger.dev.storj.io/trace/f45800a7d94cace FAIL (2025-12-26 12:53:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531210682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....] duration: 7.740231645s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:52:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 7.740263296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................X....................X.XXX...X..X.] duration: 55.788700899s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 55.788726758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.....X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX] duration: 5.00134596s trace: https://jaeger.dev.storj.io/trace/f45800a7d94cace FAIL (2025-12-26 12:52:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001365898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX....] duration: 5.710693182s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:52:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 5.710738257s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXX] duration: 5.002336555s trace: https://jaeger.dev.storj.io/trace/598a6fad3a038a16 FAIL (2025-12-26 12:52:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002363275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X....................X.XXX...X..X] duration: 1m0.003064344s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:51:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 1m0.003088571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX...] duration: 5.75897663s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:51:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 5.759006296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XX] duration: 5.001232943s trace: https://jaeger.dev.storj.io/trace/319d2c96b51928b6 FAIL (2025-12-26 12:51:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001257019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.X] duration: 5.001428356s trace: https://jaeger.dev.storj.io/trace/60759934fa4193c7 FAIL (2025-12-26 12:51:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001447492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.] duration: 5.000442548s trace: https://jaeger.dev.storj.io/trace/364fd33ca62f4be4 FAIL (2025-12-26 12:50:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000465381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX..] duration: 6.260725515s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:50:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 6.260761415s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)