Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
9.061323599s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:56:23 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 9.061348467s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [.X..............XX...X......X..X..X.....X.....X....XXX...XXX] duration:
5.001211844s trace:
https://jaeger.dev.storj.io/trace/7ebc9b94972b56d1
FAIL (2025-12-30 06:55:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001237723s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
9.832260336s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:55:44 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 9.832284243s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X............X..........................................] duration:
47.376253197s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:55:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 47.376340223s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
7.23656254s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:55:34 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 7.236597276s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [..X..............XX...X......X..X..X.....X.....X....XXX...XX] duration:
5.001606936s trace:
https://jaeger.dev.storj.io/trace/5210f969e209b86e
FAIL (2025-12-30 06:55:34 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001628907s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X....X...X..X.X.X.....X.......X.X.....X....XX...X.....X....] duration:
5.783508991s trace:
https://jaeger.dev.storj.io/trace/19b4fc724d352610
FAIL (2025-12-30 06:55:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:51:18.192429858 (591h22m18.063512893s) [upload=518433]: Starting upload
Dec 30 06:51:18.192517335 (591h22m18.06360039s) [upload=518433]: Done starting upload
Dec 30 06:51:18.192753444 (591h22m18.063836479s) [upload=518433]: Uploading segments...
Dec 30 06:51:18.192805393 (591h22m18.063888428s) [upload=518433]: Got next segment. Inline: false
Dec 30 06:51:18.192843015 (591h22m18.06392606s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:51:18.192848635 (591h22m18.06393168s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:51:18.441082662 (591h22m18.312165697s) [seg_pos={0 0}, upload=518433]: Begin upload segment...
Dec 30 06:51:18.441140843 (591h22m18.312223878s) [seg_pos={0 0}, upload=518433]: Done begin upload segment.
Dec 30 06:51:18.441194054 (591h22m18.312277089s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441234411 (591h22m18.312317447s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441313071 (591h22m18.312396116s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441672555 (591h22m18.3127556s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441975873 (591h22m18.313058908s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442353112 (591h22m18.313436147s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442546860 (591h22m18.313629895s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442619789 (591h22m18.313702824s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442896736 (591h22m18.313979771s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443178583 (591h22m18.314261628s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443447906 (591h22m18.314530941s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443732718 (591h22m18.314815753s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444004616 (591h22m18.315087651s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444108944 (591h22m18.31519198s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444411311 (591h22m18.315494346s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444768961 (591h22m18.315851996s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:20.144385730 (591h22m20.015468775s) [upload=518433]: Next returned nil segment
Dec 30 06:51:20.144398453 (591h22m20.015481489s) [upload=518433]: Waiting for error group managing segments...
Dec 30 06:51:20.428763149 (591h22m20.299846184s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.434937806 (591h22m20.306020841s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.447711049 (591h22m20.318794104s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.450940709 (591h22m20.322023754s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455089278 (591h22m20.326172313s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455720039 (591h22m20.326803084s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.461673275 (591h22m20.33275631s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.464766646 (591h22m20.335849681s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.511371859 (591h22m20.382454894s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.516754569 (591h22m20.387837624s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.618736729 (591h22m20.489819764s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.461835552 (591h22m21.332918598s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749178138 (591h22m21.620261193s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749205550 (591h22m21.620288595s) [seg_pos={0 0}, upload=518433]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:51:21.749486766 (591h22m21.620569801s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749495843 (591h22m21.620578878s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749624508 (591h22m21.620707553s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749648523 (591h22m21.620731568s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749788840 (591h22m21.620871885s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749837013 (591h22m21.620920058s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749856900 (591h22m21.620939945s) [seg_pos={0 0}, upload=518433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:51:21.749927424 (591h22m21.621010459s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:51:21.749932033 (591h22m21.621015078s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:51:22.045134906 (591h22m21.916217941s) [upload=518433]: Done uploading segments...
Dec 30 06:51:22.045156437 (591h22m21.916239472s) [upload=518433]: Upload finished. err: <nil>
Duration: 5.783526534s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
7.869013958s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:55:07 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 7.869076096s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.653306436s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:54:57 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.653334918s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [...X..............XX...X......X..X..X.....X.....X....XXX...X] duration:
5.001466072s trace:
https://jaeger.dev.storj.io/trace/3c5c5b28aaa11661
FAIL (2025-12-30 06:54:47 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001490248s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X............X.........................................] duration:
15.350063569s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:54:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 15.350077987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
21.904550122s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:54:35 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 21.904576213s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X............X........................................] duration:
49.271407186s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:54:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 49.271434758s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X....X...X..X.X.X.....X.......X.X.....X....XX...X.....X...] duration:
5.698319237s trace:
https://jaeger.dev.storj.io/trace/19b4fc724d352610
FAIL (2025-12-30 06:54:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:51:18.192429858 (591h22m18.063512893s) [upload=518433]: Starting upload
Dec 30 06:51:18.192517335 (591h22m18.06360039s) [upload=518433]: Done starting upload
Dec 30 06:51:18.192753444 (591h22m18.063836479s) [upload=518433]: Uploading segments...
Dec 30 06:51:18.192805393 (591h22m18.063888428s) [upload=518433]: Got next segment. Inline: false
Dec 30 06:51:18.192843015 (591h22m18.06392606s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:51:18.192848635 (591h22m18.06393168s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:51:18.441082662 (591h22m18.312165697s) [seg_pos={0 0}, upload=518433]: Begin upload segment...
Dec 30 06:51:18.441140843 (591h22m18.312223878s) [seg_pos={0 0}, upload=518433]: Done begin upload segment.
Dec 30 06:51:18.441194054 (591h22m18.312277089s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441234411 (591h22m18.312317447s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441313071 (591h22m18.312396116s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441672555 (591h22m18.3127556s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441975873 (591h22m18.313058908s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442353112 (591h22m18.313436147s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442546860 (591h22m18.313629895s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442619789 (591h22m18.313702824s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442896736 (591h22m18.313979771s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443178583 (591h22m18.314261628s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443447906 (591h22m18.314530941s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443732718 (591h22m18.314815753s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444004616 (591h22m18.315087651s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444108944 (591h22m18.31519198s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444411311 (591h22m18.315494346s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444768961 (591h22m18.315851996s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:20.144385730 (591h22m20.015468775s) [upload=518433]: Next returned nil segment
Dec 30 06:51:20.144398453 (591h22m20.015481489s) [upload=518433]: Waiting for error group managing segments...
Dec 30 06:51:20.428763149 (591h22m20.299846184s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.434937806 (591h22m20.306020841s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.447711049 (591h22m20.318794104s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.450940709 (591h22m20.322023754s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455089278 (591h22m20.326172313s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455720039 (591h22m20.326803084s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.461673275 (591h22m20.33275631s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.464766646 (591h22m20.335849681s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.511371859 (591h22m20.382454894s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.516754569 (591h22m20.387837624s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.618736729 (591h22m20.489819764s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.461835552 (591h22m21.332918598s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749178138 (591h22m21.620261193s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749205550 (591h22m21.620288595s) [seg_pos={0 0}, upload=518433]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:51:21.749486766 (591h22m21.620569801s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749495843 (591h22m21.620578878s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749624508 (591h22m21.620707553s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749648523 (591h22m21.620731568s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749788840 (591h22m21.620871885s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749837013 (591h22m21.620920058s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749856900 (591h22m21.620939945s) [seg_pos={0 0}, upload=518433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:51:21.749927424 (591h22m21.621010459s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:51:21.749932033 (591h22m21.621015078s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:51:22.045134906 (591h22m21.916217941s) [upload=518433]: Done uploading segments...
Dec 30 06:51:22.045156437 (591h22m21.916239472s) [upload=518433]: Upload finished. err: <nil>
Duration: 5.698344776s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.606894588s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:54:00 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.606920739s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [....X..............XX...X......X..X..X.....X.....X....XXX...] duration:
5.001205452s trace:
https://jaeger.dev.storj.io/trace/74fcebd972f21d4
FAIL (2025-12-30 06:53:56 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001231482s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
11.376206739s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:53:54 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 11.376235955s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.731196919s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:53:31 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.731216797s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [.....X..............XX...X......X..X..X.....X.....X....XXX..] duration:
5.001376945s trace:
https://jaeger.dev.storj.io/trace/74fcebd972f21d4
FAIL (2025-12-30 06:53:21 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001404298s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X....X...X..X.X.X.....X.......X.X.....X....XX...X.....X..] duration:
6.011733216s trace:
https://jaeger.dev.storj.io/trace/19b4fc724d352610
FAIL (2025-12-30 06:53:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:51:18.192429858 (591h22m18.063512893s) [upload=518433]: Starting upload
Dec 30 06:51:18.192517335 (591h22m18.06360039s) [upload=518433]: Done starting upload
Dec 30 06:51:18.192753444 (591h22m18.063836479s) [upload=518433]: Uploading segments...
Dec 30 06:51:18.192805393 (591h22m18.063888428s) [upload=518433]: Got next segment. Inline: false
Dec 30 06:51:18.192843015 (591h22m18.06392606s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:51:18.192848635 (591h22m18.06393168s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:51:18.441082662 (591h22m18.312165697s) [seg_pos={0 0}, upload=518433]: Begin upload segment...
Dec 30 06:51:18.441140843 (591h22m18.312223878s) [seg_pos={0 0}, upload=518433]: Done begin upload segment.
Dec 30 06:51:18.441194054 (591h22m18.312277089s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441234411 (591h22m18.312317447s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441313071 (591h22m18.312396116s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441672555 (591h22m18.3127556s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441975873 (591h22m18.313058908s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442353112 (591h22m18.313436147s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442546860 (591h22m18.313629895s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442619789 (591h22m18.313702824s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442896736 (591h22m18.313979771s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443178583 (591h22m18.314261628s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443447906 (591h22m18.314530941s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443732718 (591h22m18.314815753s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444004616 (591h22m18.315087651s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444108944 (591h22m18.31519198s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444411311 (591h22m18.315494346s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444768961 (591h22m18.315851996s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:20.144385730 (591h22m20.015468775s) [upload=518433]: Next returned nil segment
Dec 30 06:51:20.144398453 (591h22m20.015481489s) [upload=518433]: Waiting for error group managing segments...
Dec 30 06:51:20.428763149 (591h22m20.299846184s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.434937806 (591h22m20.306020841s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.447711049 (591h22m20.318794104s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.450940709 (591h22m20.322023754s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455089278 (591h22m20.326172313s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455720039 (591h22m20.326803084s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.461673275 (591h22m20.33275631s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.464766646 (591h22m20.335849681s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.511371859 (591h22m20.382454894s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.516754569 (591h22m20.387837624s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.618736729 (591h22m20.489819764s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.461835552 (591h22m21.332918598s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749178138 (591h22m21.620261193s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749205550 (591h22m21.620288595s) [seg_pos={0 0}, upload=518433]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:51:21.749486766 (591h22m21.620569801s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749495843 (591h22m21.620578878s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749624508 (591h22m21.620707553s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749648523 (591h22m21.620731568s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749788840 (591h22m21.620871885s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749837013 (591h22m21.620920058s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749856900 (591h22m21.620939945s) [seg_pos={0 0}, upload=518433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:51:21.749927424 (591h22m21.621010459s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:51:21.749932033 (591h22m21.621015078s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:51:22.045134906 (591h22m21.916217941s) [upload=518433]: Done uploading segments...
Dec 30 06:51:22.045156437 (591h22m21.916239472s) [upload=518433]: Upload finished. err: <nil>
Duration: 6.011754889s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [X......X............X.......................................] duration:
37.519945267s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:53:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 37.519964605s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X....X...X..X.X.X.....X.......X.X.....X....XX...X.....X.] duration:
24.105942104s trace:
https://jaeger.dev.storj.io/trace/19b4fc724d352610
FAIL (2025-12-30 06:52:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:51:18.192429858 (591h22m18.063512893s) [upload=518433]: Starting upload
Dec 30 06:51:18.192517335 (591h22m18.06360039s) [upload=518433]: Done starting upload
Dec 30 06:51:18.192753444 (591h22m18.063836479s) [upload=518433]: Uploading segments...
Dec 30 06:51:18.192805393 (591h22m18.063888428s) [upload=518433]: Got next segment. Inline: false
Dec 30 06:51:18.192843015 (591h22m18.06392606s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:51:18.192848635 (591h22m18.06393168s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:51:18.441082662 (591h22m18.312165697s) [seg_pos={0 0}, upload=518433]: Begin upload segment...
Dec 30 06:51:18.441140843 (591h22m18.312223878s) [seg_pos={0 0}, upload=518433]: Done begin upload segment.
Dec 30 06:51:18.441194054 (591h22m18.312277089s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441234411 (591h22m18.312317447s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441313071 (591h22m18.312396116s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441672555 (591h22m18.3127556s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441975873 (591h22m18.313058908s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442353112 (591h22m18.313436147s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442546860 (591h22m18.313629895s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442619789 (591h22m18.313702824s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442896736 (591h22m18.313979771s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443178583 (591h22m18.314261628s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443447906 (591h22m18.314530941s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443732718 (591h22m18.314815753s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444004616 (591h22m18.315087651s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444108944 (591h22m18.31519198s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444411311 (591h22m18.315494346s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444768961 (591h22m18.315851996s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:20.144385730 (591h22m20.015468775s) [upload=518433]: Next returned nil segment
Dec 30 06:51:20.144398453 (591h22m20.015481489s) [upload=518433]: Waiting for error group managing segments...
Dec 30 06:51:20.428763149 (591h22m20.299846184s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.434937806 (591h22m20.306020841s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.447711049 (591h22m20.318794104s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.450940709 (591h22m20.322023754s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455089278 (591h22m20.326172313s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455720039 (591h22m20.326803084s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.461673275 (591h22m20.33275631s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.464766646 (591h22m20.335849681s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.511371859 (591h22m20.382454894s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.516754569 (591h22m20.387837624s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.618736729 (591h22m20.489819764s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.461835552 (591h22m21.332918598s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749178138 (591h22m21.620261193s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749205550 (591h22m21.620288595s) [seg_pos={0 0}, upload=518433]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:51:21.749486766 (591h22m21.620569801s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749495843 (591h22m21.620578878s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749624508 (591h22m21.620707553s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749648523 (591h22m21.620731568s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749788840 (591h22m21.620871885s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749837013 (591h22m21.620920058s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749856900 (591h22m21.620939945s) [seg_pos={0 0}, upload=518433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:51:21.749927424 (591h22m21.621010459s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:51:21.749932033 (591h22m21.621015078s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:51:22.045134906 (591h22m21.916217941s) [upload=518433]: Done uploading segments...
Dec 30 06:51:22.045156437 (591h22m21.916239472s) [upload=518433]: Upload finished. err: <nil>
Duration: 24.105968916s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.231256246s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:52:39 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.231288006s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
6.333044123s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:52:31 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.333073609s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
15.717107491s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:52:23 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 15.717130985s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X......X............X......................................] duration:
37.968837771s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:52:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 37.968853402s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [......X..............XX...X......X..X..X.....X.....X....XXX.] duration:
3.627660798s trace:
https://jaeger.dev.storj.io/trace/74fcebd972f21d4
FAIL (2025-12-30 06:52:20 UTC)
orbiter warning: context deadline exceeded
Duration: 3.627684916s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
8.424657266s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:51:59 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 8.424683668s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.....X....X...X..X.X.X.....X.......X.X.....X....XX...X.....X] duration:
30.003177392s trace:
https://jaeger.dev.storj.io/trace/19b4fc724d352610
FAIL (2025-12-30 06:51:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:51:18.192429858 (591h22m18.063512893s) [upload=518433]: Starting upload
Dec 30 06:51:18.192517335 (591h22m18.06360039s) [upload=518433]: Done starting upload
Dec 30 06:51:18.192753444 (591h22m18.063836479s) [upload=518433]: Uploading segments...
Dec 30 06:51:18.192805393 (591h22m18.063888428s) [upload=518433]: Got next segment. Inline: false
Dec 30 06:51:18.192843015 (591h22m18.06392606s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:51:18.192848635 (591h22m18.06393168s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:51:18.441082662 (591h22m18.312165697s) [seg_pos={0 0}, upload=518433]: Begin upload segment...
Dec 30 06:51:18.441140843 (591h22m18.312223878s) [seg_pos={0 0}, upload=518433]: Done begin upload segment.
Dec 30 06:51:18.441194054 (591h22m18.312277089s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441234411 (591h22m18.312317447s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441313071 (591h22m18.312396116s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441672555 (591h22m18.3127556s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.441975873 (591h22m18.313058908s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442353112 (591h22m18.313436147s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442546860 (591h22m18.313629895s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442619789 (591h22m18.313702824s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.442896736 (591h22m18.313979771s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443178583 (591h22m18.314261628s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443447906 (591h22m18.314530941s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.443732718 (591h22m18.314815753s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444004616 (591h22m18.315087651s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444108944 (591h22m18.31519198s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444411311 (591h22m18.315494346s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:18.444768961 (591h22m18.315851996s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Uploading piece...
Dec 30 06:51:20.144385730 (591h22m20.015468775s) [upload=518433]: Next returned nil segment
Dec 30 06:51:20.144398453 (591h22m20.015481489s) [upload=518433]: Waiting for error group managing segments...
Dec 30 06:51:20.428763149 (591h22m20.299846184s) [noise=true , address=145.79.227.42:4123 , piece_id=EJMB77JZVCSNL3G2QSHZVXFMTQWI4TCJNDC6IZQFDHW3XDEZ4RYQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.434937806 (591h22m20.306020841s) [noise=true , address=145.79.227.3:4149 , piece_id=5XVJZDWDVJY47ZTK6GSTGPGLSVL2JBJ7ABKCX4O43YL2PIXH5TCQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.447711049 (591h22m20.318794104s) [noise=true , address=145.79.227.12:4767 , piece_id=VMKXAWOLWKLZK2OMNNKEEZPMCC37G4GCBTMCFGF5ACU6OADR5OKQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.450940709 (591h22m20.322023754s) [noise=true , address=145.79.227.9:4366 , piece_id=WQP5O7WCS4ZB4BRSPGKCHQ5QLUIWTDOIIR3SQQGGOX6N4Q3XUG4Q, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455089278 (591h22m20.326172313s) [noise=true , address=145.79.227.8:4329 , piece_id=XF5GEZT7XEFNWGMU423HWGZC2CN4RFOJU7SS5LQBU2PB4NGQVK2A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.455720039 (591h22m20.326803084s) [noise=true , address=145.79.227.2:4115 , piece_id=V2KAZJOQXOPCN66LF4H7TML6SX5UNIULJP3EI2LUKPIF6OQMOUOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.461673275 (591h22m20.33275631s) [noise=true , address=145.79.227.48:4303 , piece_id=H6CGK7OZ7YBETKTT6KWEO7G6ZSS2GHFHGUJCNDIGV252EZTWDLOA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.464766646 (591h22m20.335849681s) [noise=true , address=145.79.227.4:4187 , piece_id=EZERXE27WKQA37V2VISBOXHX75TVMU4VBOPHVSSR5HVT5QDLCFCA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.511371859 (591h22m20.382454894s) [noise=true , address=145.79.227.25:4619 , piece_id=DIAYVXFJ3OMG3TNMEYWGM6SATQ7LZHHMIL4KPFYASOCYTTI44BWQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.516754569 (591h22m20.387837624s) [noise=true , address=145.79.227.51:4409 , piece_id=GI3DP45U4KPPTDQLWI3D5OA6ZAMCLB34LBFTDPRCRJS7DMXC6M7A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:20.618736729 (591h22m20.489819764s) [noise=true , address=77.90.31.17:4773 , piece_id=BE4WNJKAGSS23FJBAHSF75NL55LKGGRCJN6QYWASVRVZU5XQ4IBA, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.461835552 (591h22m21.332918598s) [noise=true , address=77.90.31.14:4661 , piece_id=2ONCYTMQOGAQRLCG2TI4X6HJKWSEDANYPNRAMQN2JI2DNQDKC4MQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749178138 (591h22m21.620261193s) [noise=true , address=145.79.227.44:4194 , piece_id=IWRWRAI3CI5EQCFPFEWPJELGOA6BSXXUPSPVJLXB4UOXQDVQ3RDQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: <nil>
Dec 30 06:51:21.749205550 (591h22m21.620288595s) [seg_pos={0 0}, upload=518433]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:51:21.749486766 (591h22m21.620569801s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749495843 (591h22m21.620578878s) [noise=true , address=45.91.116.10:4082 , piece_id=BJ3P7DZFMWRZKF55QQFCQTXUC2RU3GBBPJG5UBVFUIIEPBW3C7EQ, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749624508 (591h22m21.620707553s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749648523 (591h22m21.620731568s) [noise=true , address=45.91.116.7:4012 , piece_id=GHYBO47KHEYXA4MOCDZT2KXHGLSPMCGVUWKWYIIDJBVDPDZRLZ5A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749788840 (591h22m21.620871885s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 06:51:21.749837013 (591h22m21.620920058s) [noise=true , address=145.79.227.34:4906 , piece_id=627X3C4LH6BA6XS7GCPG5MXUGQEO2KY6P54VP4ZNWD3RLWWH777A, seg_pos={0 0}, upload=518433]: Long tail cancellation due to optimal threshold
Dec 30 06:51:21.749856900 (591h22m21.620939945s) [seg_pos={0 0}, upload=518433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:51:21.749927424 (591h22m21.621010459s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:51:21.749932033 (591h22m21.621015078s) [upload=518433]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:51:22.045134906 (591h22m21.916217941s) [upload=518433]: Done uploading segments...
Dec 30 06:51:22.045156437 (591h22m21.916239472s) [upload=518433]: Upload finished. err: <nil>
Duration: 30.003199866s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X......X............X.....................................] duration:
56.948529537s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:51:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 56.948547712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.74630092s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:51:43 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.746326166s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [.......X..............XX...X......X..X..X.....X.....X....XXX] duration:
5.00090861s trace:
https://jaeger.dev.storj.io/trace/74fcebd972f21d4
FAIL (2025-12-30 06:51:41 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00093042s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
11 out of 60 is failed (threshold=6) [........X..............XX...X......X..X..X.....X.....X....XX] duration:
5.001784673s trace:
https://jaeger.dev.storj.io/trace/21b1896b9001ba43
FAIL (2025-12-30 06:51:17 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001808009s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
13.450473397s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:51:06 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 13.45049591s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [......X....X...X..X.X.X.....X.......X.X.....X....XX...X.....] duration:
6.135854127s trace:
https://jaeger.dev.storj.io/trace/25efa104cb3576c8
FAIL (2025-12-30 06:50:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ) context deadline exceeded
======LOGS======
30 06:47:33.902700925 (591h18m33.77378396s) [upload=518373]: Starting upload
Dec 30 06:47:33.902775336 (591h18m33.773858372s) [upload=518373]: Done starting upload
Dec 30 06:47:33.902944639 (591h18m33.774027674s) [upload=518373]: Uploading segments...
Dec 30 06:47:33.903006506 (591h18m33.774089541s) [upload=518373]: Got next segment. Inline: false
Dec 30 06:47:33.903036744 (591h18m33.774119779s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:47:33.903043197 (591h18m33.774126232s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:47:34.148006034 (591h18m34.019089069s) [seg_pos={0 0}, upload=518373]: Begin upload segment...
Dec 30 06:47:34.148062040 (591h18m34.019145075s) [seg_pos={0 0}, upload=518373]: Done begin upload segment.
Dec 30 06:47:34.148107336 (591h18m34.019190371s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148155579 (591h18m34.019238624s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148683915 (591h18m34.01976695s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148742016 (591h18m34.019825051s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149083746 (591h18m34.020166781s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149391432 (591h18m34.020474477s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149687105 (591h18m34.02077014s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149987156 (591h18m34.021070191s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150321904 (591h18m34.021404949s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150333786 (591h18m34.021416821s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150579524 (591h18m34.021662559s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150705945 (591h18m34.02178898s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151198954 (591h18m34.022281989s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151603264 (591h18m34.022686299s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151984440 (591h18m34.023067485s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.152395842 (591h18m34.023478877s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:35.844170154 (591h18m35.715253189s) [upload=518373]: Next returned nil segment
Dec 30 06:47:35.844218737 (591h18m35.715301792s) [upload=518373]: Waiting for error group managing segments...
Dec 30 06:47:36.127146455 (591h18m35.9982295s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.130047630 (591h18m36.001130665s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.140822898 (591h18m36.011905933s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.141865904 (591h18m36.012948949s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.142992949 (591h18m36.014075984s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.154997809 (591h18m36.026080844s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.167251051 (591h18m36.038334096s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.171358592 (591h18m36.042441637s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.204427992 (591h18m36.075511027s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.219970897 (591h18m36.091053932s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.460845771 (591h18m36.331928816s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.559892892 (591h18m36.430975927s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564095394 (591h18m36.435178429s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564160248 (591h18m36.435243283s) [seg_pos={0 0}, upload=518373]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:47:36.564563686 (591h18m36.435646731s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection
Dec 30 06:47:36.564574417 (591h18m36.435657452s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.564877924 (591h18m36.435960959s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.564886651 (591h18m36.435969696s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565006689 (591h18m36.436089734s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.565014714 (591h18m36.436097749s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565048769 (591h18m36.436131804s) [seg_pos={0 0}, upload=518373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:47:36.565134443 (591h18m36.436217478s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:47:36.565140214 (591h18m36.436223259s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:47:36.856746186 (591h18m36.727829231s) [upload=518373]: Done uploading segments...
Dec 30 06:47:36.856766275 (591h18m36.72784932s) [upload=518373]: Upload finished. err: <nil>
Duration: 6.135886249s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.721791465s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:50:48 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.721824137s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X......X............X....................................] duration:
39.572391982s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:50:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 39.572423822s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [.........X..............XX...X......X..X..X.....X.....X....X] duration:
5.001696086s trace:
https://jaeger.dev.storj.io/trace/86dc469422a898c
FAIL (2025-12-30 06:50:46 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001721836s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
7.278921982s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:50:34 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.278947471s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X......X....X...X..X.X.X.....X.......X.X.....X....XX...X....] duration:
5.661635567s trace:
https://jaeger.dev.storj.io/trace/25efa104cb3576c8
FAIL (2025-12-30 06:50:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ) context deadline exceeded
======LOGS======
30 06:47:33.902700925 (591h18m33.77378396s) [upload=518373]: Starting upload
Dec 30 06:47:33.902775336 (591h18m33.773858372s) [upload=518373]: Done starting upload
Dec 30 06:47:33.902944639 (591h18m33.774027674s) [upload=518373]: Uploading segments...
Dec 30 06:47:33.903006506 (591h18m33.774089541s) [upload=518373]: Got next segment. Inline: false
Dec 30 06:47:33.903036744 (591h18m33.774119779s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:47:33.903043197 (591h18m33.774126232s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:47:34.148006034 (591h18m34.019089069s) [seg_pos={0 0}, upload=518373]: Begin upload segment...
Dec 30 06:47:34.148062040 (591h18m34.019145075s) [seg_pos={0 0}, upload=518373]: Done begin upload segment.
Dec 30 06:47:34.148107336 (591h18m34.019190371s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148155579 (591h18m34.019238624s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148683915 (591h18m34.01976695s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148742016 (591h18m34.019825051s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149083746 (591h18m34.020166781s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149391432 (591h18m34.020474477s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149687105 (591h18m34.02077014s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149987156 (591h18m34.021070191s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150321904 (591h18m34.021404949s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150333786 (591h18m34.021416821s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150579524 (591h18m34.021662559s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150705945 (591h18m34.02178898s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151198954 (591h18m34.022281989s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151603264 (591h18m34.022686299s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151984440 (591h18m34.023067485s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.152395842 (591h18m34.023478877s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:35.844170154 (591h18m35.715253189s) [upload=518373]: Next returned nil segment
Dec 30 06:47:35.844218737 (591h18m35.715301792s) [upload=518373]: Waiting for error group managing segments...
Dec 30 06:47:36.127146455 (591h18m35.9982295s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.130047630 (591h18m36.001130665s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.140822898 (591h18m36.011905933s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.141865904 (591h18m36.012948949s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.142992949 (591h18m36.014075984s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.154997809 (591h18m36.026080844s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.167251051 (591h18m36.038334096s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.171358592 (591h18m36.042441637s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.204427992 (591h18m36.075511027s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.219970897 (591h18m36.091053932s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.460845771 (591h18m36.331928816s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.559892892 (591h18m36.430975927s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564095394 (591h18m36.435178429s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564160248 (591h18m36.435243283s) [seg_pos={0 0}, upload=518373]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:47:36.564563686 (591h18m36.435646731s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection
Dec 30 06:47:36.564574417 (591h18m36.435657452s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.564877924 (591h18m36.435960959s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.564886651 (591h18m36.435969696s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565006689 (591h18m36.436089734s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.565014714 (591h18m36.436097749s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565048769 (591h18m36.436131804s) [seg_pos={0 0}, upload=518373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:47:36.565134443 (591h18m36.436217478s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:47:36.565140214 (591h18m36.436223259s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:47:36.856746186 (591h18m36.727829231s) [upload=518373]: Done uploading segments...
Dec 30 06:47:36.856766275 (591h18m36.72784932s) [upload=518373]: Upload finished. err: <nil>
Duration: 5.661651748s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X......X....X...X..X.X.X.....X.......X.X.....X....XX...X...] duration:
30.002336918s trace:
https://jaeger.dev.storj.io/trace/25efa104cb3576c8
FAIL (2025-12-30 06:50:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ) context deadline exceeded
======LOGS======
30 06:47:33.902700925 (591h18m33.77378396s) [upload=518373]: Starting upload
Dec 30 06:47:33.902775336 (591h18m33.773858372s) [upload=518373]: Done starting upload
Dec 30 06:47:33.902944639 (591h18m33.774027674s) [upload=518373]: Uploading segments...
Dec 30 06:47:33.903006506 (591h18m33.774089541s) [upload=518373]: Got next segment. Inline: false
Dec 30 06:47:33.903036744 (591h18m33.774119779s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:47:33.903043197 (591h18m33.774126232s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:47:34.148006034 (591h18m34.019089069s) [seg_pos={0 0}, upload=518373]: Begin upload segment...
Dec 30 06:47:34.148062040 (591h18m34.019145075s) [seg_pos={0 0}, upload=518373]: Done begin upload segment.
Dec 30 06:47:34.148107336 (591h18m34.019190371s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148155579 (591h18m34.019238624s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148683915 (591h18m34.01976695s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148742016 (591h18m34.019825051s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149083746 (591h18m34.020166781s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149391432 (591h18m34.020474477s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149687105 (591h18m34.02077014s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149987156 (591h18m34.021070191s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150321904 (591h18m34.021404949s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150333786 (591h18m34.021416821s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150579524 (591h18m34.021662559s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150705945 (591h18m34.02178898s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151198954 (591h18m34.022281989s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151603264 (591h18m34.022686299s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151984440 (591h18m34.023067485s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.152395842 (591h18m34.023478877s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:35.844170154 (591h18m35.715253189s) [upload=518373]: Next returned nil segment
Dec 30 06:47:35.844218737 (591h18m35.715301792s) [upload=518373]: Waiting for error group managing segments...
Dec 30 06:47:36.127146455 (591h18m35.9982295s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.130047630 (591h18m36.001130665s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.140822898 (591h18m36.011905933s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.141865904 (591h18m36.012948949s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.142992949 (591h18m36.014075984s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.154997809 (591h18m36.026080844s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.167251051 (591h18m36.038334096s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.171358592 (591h18m36.042441637s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.204427992 (591h18m36.075511027s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.219970897 (591h18m36.091053932s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.460845771 (591h18m36.331928816s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.559892892 (591h18m36.430975927s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564095394 (591h18m36.435178429s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564160248 (591h18m36.435243283s) [seg_pos={0 0}, upload=518373]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:47:36.564563686 (591h18m36.435646731s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection
Dec 30 06:47:36.564574417 (591h18m36.435657452s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.564877924 (591h18m36.435960959s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.564886651 (591h18m36.435969696s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565006689 (591h18m36.436089734s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.565014714 (591h18m36.436097749s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565048769 (591h18m36.436131804s) [seg_pos={0 0}, upload=518373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:47:36.565134443 (591h18m36.436217478s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:47:36.565140214 (591h18m36.436223259s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:47:36.856746186 (591h18m36.727829231s) [upload=518373]: Done uploading segments...
Dec 30 06:47:36.856766275 (591h18m36.72784932s) [upload=518373]: Upload finished. err: <nil>
Duration: 30.002365503s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [..........X..............XX...X......X..X..X.....X.....X....] duration:
3.547380863s trace:
https://jaeger.dev.storj.io/trace/2dfc5fc3a860afad
FAIL (2025-12-30 06:50:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.547406192s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.902045153s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:50:02 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.902076543s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X...X......X............X...................................] duration:
14.628108321s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:49:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 14.628139189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
9.069627688s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:49:56 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 9.069696419s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [...........X..............XX...X......X..X..X.....X.....X...] duration:
3.373867784s trace:
https://jaeger.dev.storj.io/trace/2dfc5fc3a860afad
FAIL (2025-12-30 06:49:41 UTC)
orbiter warning: context deadline exceeded
Duration: 3.373895588s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X...X......X............X..................................] duration:
42.349998211s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:49:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 42.350030802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.859315229s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:49:33 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 5.859334174s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [X...........X..............XX...X......X..X..X.....X.....X..] duration:
3.620341681s trace:
https://jaeger.dev.storj.io/trace/2dfc5fc3a860afad
FAIL (2025-12-30 06:49:10 UTC)
orbiter warning: context deadline exceeded
Duration: 3.620376196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
7.07657995s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:49:04 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 7.076617763s
core
uplink
eu-select-1
eu1-placement-15
|