Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [.X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X..] duration:
3.506504047s trace:
https://jaeger.dev.storj.io/trace/42949bd00e591a18
FAIL (2025-12-08 16:02:01 UTC)
orbiter warning: context deadline exceeded
Duration: 3.506530439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X...........] duration:
8.543481983s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:01:58 UTC)
orbiter warning: context deadline exceeded
Duration: 8.543509005s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.824250289s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:01:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.824277673s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........X.........................X..X...X.................X] duration:
1m0.001324703s trace:
https://jaeger.dev.storj.io/trace/59a6790b91233307
FAIL (2025-12-08 16:01:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001360681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.009510027s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:01:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.009540944s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.577307816s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:01:43 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.577334327s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X.........X...X..............X.............X............] duration:
6.418417588s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:01:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.418436695s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X..........] duration:
9.443590854s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:01:37 UTC)
orbiter warning: context deadline exceeded
Duration: 9.443618428s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X.........X...X..............X.............X...........] duration:
7.634677435s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:01:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 7.634706701s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
7.124277289s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:01:14 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.124311583s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001489626s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:01:11 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.001514995s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
10.308565724s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:01:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 10.308665844s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [..X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.] duration:
3.385234085s trace:
https://jaeger.dev.storj.io/trace/42949bd00e591a18
FAIL (2025-12-08 16:01:07 UTC)
orbiter warning: context deadline exceeded
Duration: 3.385325381s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X.........] duration:
25.225610339s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:01:01 UTC)
orbiter warning: context deadline exceeded
Duration: 25.225680352s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [...X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X] duration:
5.001758089s trace:
https://jaeger.dev.storj.io/trace/42949bd00e591a18
FAIL (2025-12-08 16:00:35 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001808695s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.056298902s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 16:00:33 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.05633012s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X........] duration:
28.440888787s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:00:29 UTC)
orbiter warning: context deadline exceeded
Duration: 28.440918613s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.149698526s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:00:14 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.149727692s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X.........X...X..............X.............X..........] duration:
6.149771532s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:00:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.149803191s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........X.........................X..X...X.................] duration:
6.118594007s trace:
https://jaeger.dev.storj.io/trace/33a3f0149193afa6
FAIL (2025-12-08 16:00:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.118626419s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.029542761s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:00:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.029574762s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.723476305s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:59:44 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.723499138s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..........X.........................X..X...X................] duration:
5.710871496s trace:
https://jaeger.dev.storj.io/trace/33a3f0149193afa6
FAIL (2025-12-08 15:59:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.710949414s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.769824304s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:59:31 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.769846206s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X.........X...X..............X.............X.........] duration:
6.066308309s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 15:59:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.066347253s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X.......] duration:
5.99329083s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 15:59:24 UTC)
orbiter warning: context deadline exceeded
Duration: 5.993307911s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....] duration:
3.415089168s trace:
https://jaeger.dev.storj.io/trace/18158d02fe1b69d5
FAIL (2025-12-08 15:59:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.415117792s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [....X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X......] duration:
21.312735407s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 15:59:18 UTC)
orbiter warning: context deadline exceeded
Duration: 21.312762478s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X.........X...X..............X.............X........] duration:
6.504109922s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 15:59:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.504138387s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.133340331s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:59:01 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.133369919s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
6.324165353s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:59:00 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.324239453s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
6.257610008s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:58:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.257655353s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X.........X...X..............X.............X.......] duration:
6.412270315s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 15:58:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.412306733s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...........X.........................X..X...X...............] duration:
5.902004422s trace:
https://jaeger.dev.storj.io/trace/33a3f0149193afa6
FAIL (2025-12-08 15:58:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.902033698s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.119195414s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:58:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.119227595s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX....] duration:
3.491796913s trace:
https://jaeger.dev.storj.io/trace/18158d02fe1b69d5
FAIL (2025-12-08 15:58:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.491851999s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
10.359533987s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:58:01 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.359566177s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.....X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X.....] duration:
6.123071136s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 15:57:59 UTC)
orbiter warning: context deadline exceeded
Duration: 6.123103048s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........X.........X...X..............X.............X......] duration:
5.942851231s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 15:57:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 5.942872402s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.241120922s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:57:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.241153574s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.335623264s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:57:45 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.335653332s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [......X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X....] duration:
14.267807748s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 15:57:38 UTC)
orbiter warning: context deadline exceeded
Duration: 14.2681024s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [............X.........................X..X...X..............] duration:
5.652672941s trace:
https://jaeger.dev.storj.io/trace/33a3f0149193afa6
FAIL (2025-12-08 15:57:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.652702148s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X.........X...X..............X.............X.....] duration:
6.087515062s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 15:57:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.087546881s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.197672601s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:57:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.197702397s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
2.262228205s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:57:10 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 2.262251217s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [.X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX...] duration:
3.556826193s trace:
https://jaeger.dev.storj.io/trace/18158d02fe1b69d5
FAIL (2025-12-08 15:57:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.556859077s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
6.419663341s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:56:54 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.419692085s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.......X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X...] duration:
5.952082059s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 15:56:54 UTC)
orbiter warning: context deadline exceeded
Duration: 5.952106797s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X.........X...X..............X.............X....] duration:
7.579627369s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 15:56:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 7.57965917s
core
uplink
us-select-1
|