Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
1.430094545s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:13:25 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.430125705s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [..XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X..] duration:
3.507205112s trace:
https://jaeger.dev.storj.io/trace/44072b2910bf5c31
FAIL (2025-12-08 16:13:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.507232546s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X..............X.............X...........................] duration:
7.324323822s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:13:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 7.324347747s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [...XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X.] duration:
3.762580558s trace:
https://jaeger.dev.storj.io/trace/44072b2910bf5c31
FAIL (2025-12-08 16:13:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.762634581s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
4.721019103s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:13:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.721048539s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [....XX...X.X.X.X....XX......X....X..........................] duration:
6.055145075s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:13:01 UTC)
orbiter warning: context deadline exceeded
Duration: 6.055176034s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.................X..X...X.................X.X...............] duration:
5.896063812s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:13:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.896093408s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
1.411066651s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:12:56 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.411100156s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X] duration:
5.00155986s trace:
https://jaeger.dev.storj.io/trace/44072b2910bf5c31
FAIL (2025-12-08 16:12:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001688956s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X...X..............X.............X..........................] duration:
6.580765003s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:12:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.580781534s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..................X..X...X.................X.X..............] duration:
5.762794216s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:12:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.762834644s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.....XX...X.X.X.X....XX......X....X.........................] duration:
6.011079707s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:12:12 UTC)
orbiter warning: context deadline exceeded
Duration: 6.011104816s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
5.74760217s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:12:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.747635003s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..] duration:
3.516889635s trace:
https://jaeger.dev.storj.io/trace/60bfaa806499e0aa
FAIL (2025-12-08 16:11:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.516921484s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
1.417482131s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:11:48 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.417511405s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X...X..............X.............X.........................] duration:
6.75391586s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:11:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.753944905s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X.....XX...X.X.X.X....XX......X....X........................] duration:
6.106484355s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:11:10 UTC)
orbiter warning: context deadline exceeded
Duration: 6.106519483s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...................X..X...X.................X.X.............] duration:
6.020182029s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:11:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.020262632s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X...X..............X.............X........................] duration:
7.072921711s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:11:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 7.072951908s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.404635216s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:10:57 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.404678928s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX.] duration:
3.468873581s trace:
https://jaeger.dev.storj.io/trace/60bfaa806499e0aa
FAIL (2025-12-08 16:10:57 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.468901435s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
5.744874637s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:10:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.744905556s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X.....XX...X.X.X.X....XX......X....X.......................] duration:
7.350310038s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:10:53 UTC)
orbiter warning: context deadline exceeded
Duration: 7.350513335s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X...X..............X.............X.......................] duration:
7.461946492s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:10:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 7.46197596s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....................X..X...X.................X.X............] duration:
24.032898676s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:10:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.032926789s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
1.393147099s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:10:18 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.393178899s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX] duration:
5.00257218s trace:
https://jaeger.dev.storj.io/trace/60bfaa806499e0aa
FAIL (2025-12-08 16:10:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002600635s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
1.4362582s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:09:59 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.436287566s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..X.....XX...X.X.X.X....XX......X....X......................] duration:
8.324182566s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:09:44 UTC)
orbiter warning: context deadline exceeded
Duration: 8.324210611s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X...X..............X.............X......................] duration:
9.231837097s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:09:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 9.231863117s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
4.735343955s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:09:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.735374033s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..X] duration:
5.000783977s trace:
https://jaeger.dev.storj.io/trace/741ce01c6131d670
FAIL (2025-12-08 16:09:30 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000807301s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....................X..X...X.................X.X...........] duration:
6.063080489s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:09:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.063107161s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
9.234526072s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:09:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 9.234557752s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
1.429225658s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:09:13 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.429257076s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
1.577246841s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:08:55 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.577276047s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......................X..X...X.................X.X..........] duration:
11.219497397s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:08:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.219558683s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X..X.....XX...X.X.X.X....XX......X....X.....................] duration:
9.900135829s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:08:49 UTC)
orbiter warning: context deadline exceeded
Duration: 9.900162811s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
6.37467893s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:08:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.37471103s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X...X..............X.............X.....................] duration:
6.783117311s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:08:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.783150294s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..] duration:
4.625168767s trace:
https://jaeger.dev.storj.io/trace/60304221a9a65d68
FAIL (2025-12-08 16:08:26 UTC)
orbiter warning: context deadline exceeded
Duration: 4.625197863s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [XX..X.....XX...X.X.X.X....XX......X....X....................] duration:
19.817640778s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:08:15 UTC)
orbiter warning: context deadline exceeded
Duration: 19.817667257s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......................X..X...X.................X.X.........] duration:
17.11520725s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:08:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.115232207s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.XX..X.....XX...X.X.X.X....XX......X....X...................] duration:
18.117926228s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:07:55 UTC)
orbiter warning: context deadline exceeded
Duration: 18.117958561s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........................X..X...X.................X.X........] duration:
19.546347394s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:07:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.546372172s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
1.565654908s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:07:44 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.565673573s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.807422912s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:07:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.807454211s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X...X..............X.............X....................] duration:
6.164029583s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:07:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.164057797s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X.] duration:
3.448280973s trace:
https://jaeger.dev.storj.io/trace/60304221a9a65d68
FAIL (2025-12-08 16:07:23 UTC)
orbiter warning: context deadline exceeded
Duration: 3.44833681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.055745031s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:07:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.055773766s
core
uplink
eu-select-1
eu1-placement-15
|