Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.633464928s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:57:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.633481419s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.023237121s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:57:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.02328355s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X...X...............X..........................X.] duration: 3.405023345s trace: https://jaeger.dev.storj.io/trace/5e9c07f7bd8358c3 FAIL (2025-12-08 01:57:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.405118367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.638011426s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 01:57:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.638037575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.693594324s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:57:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.693616065s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 4.8188484s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 01:57:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.818878247s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.101720926s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 01:57:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.101749982s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.....X........................X................X..] duration: 5.943777085s trace: https://jaeger.dev.storj.io/trace/4277a026895ce58c FAIL (2025-12-08 01:57:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.943801331s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.841200882s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:57:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.841227293s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X...X...............X..........................X] duration: 5.001204175s trace: https://jaeger.dev.storj.io/trace/5e9c07f7bd8358c3 FAIL (2025-12-08 01:56:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001220987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.222288348s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:56:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.222316592s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.777701669s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 01:56:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.777731547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.315736409s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:56:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.315766667s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.774244857s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:56:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.774271998s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.261096253s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 01:56:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.261123506s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.774123536s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 01:56:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.774181177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.007986662s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 01:56:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.008015546s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.....X........................X................X.] duration: 6.092205649s trace: https://jaeger.dev.storj.io/trace/4277a026895ce58c FAIL (2025-12-08 01:56:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.092226879s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X...X...............X..........................] duration: 3.389584056s trace: https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f FAIL (2025-12-08 01:55:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.3896119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.....X........................X................X] duration: 1m44.245038587s trace: https://jaeger.dev.storj.io/trace/4277a026895ce58c FAIL (2025-12-08 01:55:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m44.245062763s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 9.56428388s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 01:55:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.564302064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.694535188s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:55:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.694560366s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m13.62131902s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 01:55:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m13.621340852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.845363407s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:55:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.845387503s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X...X...............X.........................] duration: 3.392533419s trace: https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f FAIL (2025-12-08 01:55:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39255386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.983693863s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 01:54:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.983728699s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.758139521s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:54:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.758166773s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.29050771s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:54:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.290536785s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X...X...............X........................] duration: 3.498932235s trace: https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f FAIL (2025-12-08 01:54:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498963755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 4.865991449s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 01:54:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.866010024s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 3m55.560108427s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 01:54:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3m55.560129486s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X...X...............X.......................] duration: 3.540292462s trace: https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f FAIL (2025-12-08 01:53:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540325776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.479978804s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:53:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.480000846s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.....X........................X................] duration: 5.9534168s trace: https://jaeger.dev.storj.io/trace/7d8de2b6e831abb5 FAIL (2025-12-08 01:53:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.953448511s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.786095012s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:53:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.786123427s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.406095988s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:53:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.406120485s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.....X........................X...............] duration: 5.94430309s trace: https://jaeger.dev.storj.io/trace/7d8de2b6e831abb5 FAIL (2025-12-08 01:53:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.94433436s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.799452979s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:52:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.799482606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X...X...............X......................] duration: 3.383467499s trace: https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f FAIL (2025-12-08 01:52:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383497236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.996970074s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:52:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.997063732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.....X........................X..............] duration: 5.931302752s trace: https://jaeger.dev.storj.io/trace/7d8de2b6e831abb5 FAIL (2025-12-08 01:52:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.931333511s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.709334908s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:52:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.709364443s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.018720913s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:52:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.018748604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.....X........................X.............] duration: 5.812286736s trace: https://jaeger.dev.storj.io/trace/7d8de2b6e831abb5 FAIL (2025-12-08 01:52:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.812310351s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.808671912s trace: https://jaeger.dev.storj.io/trace/cb4dffac862a1ed FAIL (2025-12-08 01:51:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments... Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment... Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment. Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece... Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments... Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil> Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces. Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments... Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.808696739s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X...X...............X.....................] duration: 3.42620688s trace: https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f FAIL (2025-12-08 01:51:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426461806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.....X........................X............] duration: 5.928871122s trace: https://jaeger.dev.storj.io/trace/7d8de2b6e831abb5 FAIL (2025-12-08 01:51:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.928895809s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.764825059s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 01:51:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.764853262s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.690414941s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:51:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.690490716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X...X...............X....................] duration: 3.469313836s trace: https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f FAIL (2025-12-08 01:51:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469354463s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)