Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [X.....X.....................X...X...............X...........] duration:
3.518218425s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:44:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.518242582s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.611241734s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:44:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.611295427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.079736253s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:44:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.079760039s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.....X........................X.] duration:
5.977652739s trace:
https://jaeger.dev.storj.io/trace/7d8de2b6e831abb5
FAIL (2025-12-08 01:44:09 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.977668459s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................X.....X........................X] duration:
1m8.525108057s trace:
https://jaeger.dev.storj.io/trace/7d8de2b6e831abb5
FAIL (2025-12-08 01:44:03 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1m8.525131251s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.X.....X.....................X...X...............X..........] duration:
3.548617711s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:43:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.548672204s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
5.623840331s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:43:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.62386637s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.84958145s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:43:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.849610854s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.796019448s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:43:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.796050187s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..X.....X.....................X...X...............X.........] duration:
3.411826616s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:42:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.411856354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
5.824671203s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:42:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.824696631s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X...................................X.......................] duration:
15.192195776s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:42:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 15.192227657s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.627326615s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:42:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.627353986s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...X.....X.....................X...X...............X........] duration:
3.540022963s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:42:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.54005258s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XX...................................X......................] duration:
6.661444453s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:41:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.661475853s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X.....X........................] duration:
5.763621208s trace:
https://jaeger.dev.storj.io/trace/3eaf904a46a0a5a6
FAIL (2025-12-08 01:41:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.763650864s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
5.812493449s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:41:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.812538294s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
5.612521206s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:41:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.612543109s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X.....X.......................] duration:
5.748778114s trace:
https://jaeger.dev.storj.io/trace/3eaf904a46a0a5a6
FAIL (2025-12-08 01:41:24 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.748808513s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
5.868393754s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:41:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.868419803s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [....X.....X.....................X...X...............X.......] duration:
3.388251057s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:41:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.388287806s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [X....X.....X.....................X...X...............X......] duration:
3.519712871s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:40:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.519741244s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X..............X............................................] duration:
5.695838398s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:40:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.695867725s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XX...................................X.....................] duration:
5.719943688s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:40:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.719971741s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.....X......................] duration:
5.777812679s trace:
https://jaeger.dev.storj.io/trace/3eaf904a46a0a5a6
FAIL (2025-12-08 01:40:12 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.77784917s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
5.464496432s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:40:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.464530918s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X..............X...........................................] duration:
5.839191265s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:39:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.839224018s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.X....X.....X.....................X...X...............X.....] duration:
3.392635814s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:39:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.39266472s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..XX...................................X....................] duration:
7.112492523s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:39:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 7.112531216s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X..............X..........................................] duration:
5.952906438s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:39:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.952964519s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
6.044594179s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:39:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.044618615s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...XX...................................X...................] duration:
5.98013208s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:39:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.980152538s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X.....X.....................] duration:
6.215830089s trace:
https://jaeger.dev.storj.io/trace/3eaf904a46a0a5a6
FAIL (2025-12-08 01:39:02 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.215864494s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [..X....X.....X.....................X...X...............X....] duration:
3.548693506s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:38:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.548739693s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X..............X.........................................] duration:
5.759951108s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:38:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.759982457s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [...X....X.....X.....................X...X...............X...] duration:
4.238542392s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:38:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.238568924s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
5.82941631s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:38:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.829552439s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....XX...................................X..................] duration:
6.413836607s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:38:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.413868757s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X.....X....................] duration:
6.088853033s trace:
https://jaeger.dev.storj.io/trace/3eaf904a46a0a5a6
FAIL (2025-12-08 01:38:02 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.088881788s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X..............X........................................] duration:
5.749698408s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:38:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.749857521s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [....X....X.....X.....................X...X...............X..] duration:
3.394909274s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:38:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.39493906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X.....X...................] duration:
5.767812799s trace:
https://jaeger.dev.storj.io/trace/3eaf904a46a0a5a6
FAIL (2025-12-08 01:37:40 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.76783936s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....X..............X.......................................] duration:
5.985514532s trace:
https://jaeger.dev.storj.io/trace/79dc68a558cc00ae
FAIL (2025-12-08 01:37:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.985540351s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
5.671495113s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:37:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.671545769s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....XX...................................X.................] duration:
5.883948903s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:37:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.883982197s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.....X....X.....X.....................X...X...............X.] duration:
3.426929558s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:37:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.426956148s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X.....X..................] duration:
5.944881466s trace:
https://jaeger.dev.storj.io/trace/3eaf904a46a0a5a6
FAIL (2025-12-08 01:36:47 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.944921903s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
5.792405077s trace:
https://jaeger.dev.storj.io/trace/38fc930a38463b73
FAIL (2025-12-08 01:36:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece...
Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment
Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments...
Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil>
Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces.
Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF
Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold
Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments...
Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.792429413s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [......X....X.....X.....................X...X...............X] duration:
5.001959066s trace:
https://jaeger.dev.storj.io/trace/55f5dabdd73b2e1f
FAIL (2025-12-08 01:36:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001990556s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......XX...................................X................] duration:
5.846278271s trace:
https://jaeger.dev.storj.io/trace/cb4dffac862a1ed
FAIL (2025-12-08 01:36:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 01:23:20.453899048 (57h54m20.324982083s) [upload=50875]: Starting upload
Dec 8 01:23:20.453963411 (57h54m20.325046456s) [upload=50875]: Done starting upload
Dec 8 01:23:20.454102616 (57h54m20.325185671s) [upload=50875]: Uploading segments...
Dec 8 01:23:20.454225621 (57h54m20.325308666s) [upload=50875]: Got next segment. Inline: false
Dec 8 01:23:20.454272009 (57h54m20.325355044s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 01:23:20.454313708 (57h54m20.325396743s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 01:23:20.700431195 (57h54m20.57151424s) [seg_pos={0 0}, upload=50875]: Begin upload segment...
Dec 8 01:23:20.700489777 (57h54m20.571572822s) [seg_pos={0 0}, upload=50875]: Done begin upload segment.
Dec 8 01:23:20.700528470 (57h54m20.571611516s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700563567 (57h54m20.571646612s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.700982916 (57h54m20.572065951s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701307133 (57h54m20.572390168s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701376244 (57h54m20.57245929s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701646329 (57h54m20.572729364s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701835639 (57h54m20.572918684s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.701980516 (57h54m20.573063561s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702244508 (57h54m20.573327543s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702365839 (57h54m20.573448874s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702637607 (57h54m20.573720652s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.702714333 (57h54m20.573797368s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703016578 (57h54m20.574099613s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703037398 (57h54m20.574120443s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703445816 (57h54m20.574528851s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:20.703899490 (57h54m20.574982525s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Uploading piece...
Dec 8 01:23:21.746617940 (57h54m21.617700975s) [noise=true , address=77.90.31.19:4841 , piece_id=BBISLRAB56NS7OUATVCSLUKQWIODYZ5MH7GL2RIEJR6QQM5FLBQA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 01:23:22.393983771 (57h54m22.265066806s) [upload=50875]: Next returned nil segment
Dec 8 01:23:22.394010112 (57h54m22.265093147s) [upload=50875]: Waiting for error group managing segments...
Dec 8 01:23:22.675909735 (57h54m22.54699278s) [noise=true , address=145.79.227.28:4725 , piece_id=OO5Z35B5D5VZUIUEZL6YCORIJQM5MNV27JG5SO3BZ7PN6HYMIIRQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.682375107 (57h54m22.553458142s) [noise=true , address=145.79.227.17:4970 , piece_id=GSAFFLNUUHD266TVIHREJB7N6FLCMKM7UG3D3LGH23ZHQFXVKGUQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.684169534 (57h54m22.555252569s) [noise=true , address=145.79.227.21:4473 , piece_id=6ZNPZNMNILOVA5Y7XXZV2B5X6RE7KVDPHE5GQB6BWCPYMDSSBHAA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.686983912 (57h54m22.558066957s) [noise=true , address=145.79.227.43:4158 , piece_id=NRCRCPVFTFJ2Z3ZVWAJSGRLIVJ3H2W7BVAZNYIGVQOM3VVHCSMKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687147975 (57h54m22.55823101s) [noise=true , address=145.79.227.22:4511 , piece_id=6B54KRO42XGJUVNEUSKY3H6IDUCOGFNDG3JL6XKGXEDS2IGCG3FQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.687636154 (57h54m22.558719189s) [noise=true , address=145.79.227.26:4654 , piece_id=WJJLT5NTHKROOUR6QI7TXMESKWZ7SFKUDJV3UAY3G3RX4SHEKSMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.695437690 (57h54m22.566520725s) [noise=true , address=145.79.227.38:4979 , piece_id=2V42Z5UJVM7NAYIHGNUQDKIIRJRZWF3E5WODIVT5DWG7T7ESQM4A, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.703533215 (57h54m22.574616251s) [noise=true , address=145.79.227.23:4545 , piece_id=7KCBGK7AK3GCTVPW7NBKGKXP57S2WTYH2KHBIF4BA5BJFSC43VBA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.705656759 (57h54m22.576739804s) [noise=true , address=145.79.227.41:4086 , piece_id=FADS333AVZHK6T4PP32KXASD5X74YJ6XUU3I245CV3T7EF3V6WKQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.707098905 (57h54m22.57818195s) [noise=true , address=145.79.227.34:4906 , piece_id=N4MT2F4N3CNHDEXXZSSQ2GMSWNFXTYPGK4AE665RTMHV7Y2VLTSA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.712700611 (57h54m22.583783657s) [noise=true , address=145.79.227.27:4689 , piece_id=7ORO2UXSDBBZNI6A4T2AW4Q3QIVHG7TS2GTY5HWZJEPMGGJ3CVGA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.714712362 (57h54m22.585795397s) [noise=true , address=145.79.227.26:4653 , piece_id=XOTYKAODPCRM4ICXIF6MIKBVCRO3L2DHPPDWXAHYFNDCJZ26YV5Q, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725749519 (57h54m22.596832564s) [noise=true , address=145.79.227.32:4834 , piece_id=BYIJKLYZBU3S3J4366IK5C7WKYCZTQYGAGOO6T2IIDYGK4SEDZMQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: <nil>
Dec 8 01:23:22.725769797 (57h54m22.596852842s) [seg_pos={0 0}, upload=50875]: Segment reached optimal threshold of 13 pieces.
Dec 8 01:23:22.726092312 (57h54m22.597175347s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 8 01:23:22.726101048 (57h54m22.597184083s) [noise=true , address=77.90.31.10:4517 , piece_id=66N3WMQPZZMT6BFOL4C5JKMUWT4ZZRAHDWUPCYCY4BQQXHZ2Q4GA, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726380380 (57h54m22.597463435s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:49308->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 8 01:23:22.726391471 (57h54m22.597474516s) [noise=true , address=145.79.227.25:4618 , piece_id=BJYLWAUYOLSWLUT5QHHUWEDDADJP7DUQS6XDOXJKGN7H6IEKJOOQ, seg_pos={0 0}, upload=50875]: Long tail cancellation due to optimal threshold
Dec 8 01:23:22.726411019 (57h54m22.597494064s) [seg_pos={0 0}, upload=50875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 01:23:22.726498915 (57h54m22.59758196s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 01:23:22.726506450 (57h54m22.597589495s) [upload=50875]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 01:23:55.590067327 (57h54m55.461150372s) [upload=50875]: Done uploading segments...
Dec 8 01:23:55.590116190 (57h54m55.461199225s) [upload=50875]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.846328275s
core
uplink
us-select-1
|