Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X...X..............X.............X...................] duration:
9.097498969s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:07:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 9.097530531s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [.X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X] duration:
5.001514504s trace:
https://jaeger.dev.storj.io/trace/60304221a9a65d68
FAIL (2025-12-08 16:06:54 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001533943s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........................X..X...X.................X.X.......] duration:
5.775519229s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:06:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.775548225s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X.XX..X.....XX...X.X.X.X....XX......X....X..................] duration:
17.065957055s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:06:51 UTC)
orbiter warning: context deadline exceeded
Duration: 17.065973727s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [XX.XX..X.....XX...X.X.X.X....XX......X....X.................] duration:
28.787149061s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:06:33 UTC)
orbiter warning: context deadline exceeded
Duration: 28.787178797s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
6.324095989s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:06:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.324120697s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.496532534s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:06:29 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.496562551s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [X.........................X..X...X.................X.X......] duration:
5.912721811s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:06:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.912766816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.631684512s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:06:13 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.63170983s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.X.........................X..X...X.................X.X.....] duration:
6.029881514s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:06:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.02991068s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.381322164s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:06:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.381352744s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X...X..............X.............X..................] duration:
8.084632071s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:06:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 8.084677268s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [..X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...] duration:
3.564375429s trace:
https://jaeger.dev.storj.io/trace/6d35f8d6110e790f
FAIL (2025-12-08 16:05:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.564402531s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
23.304715052s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:05:41 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 23.304743806s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X...X..............X.............X.................] duration:
6.974136159s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:05:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.974162059s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.XX.XX..X.....XX...X.X.X.X....XX......X....X................] duration:
19.21614474s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:05:38 UTC)
orbiter warning: context deadline exceeded
Duration: 19.216203673s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..X.........................X..X...X.................X.X....] duration:
5.855051076s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:05:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.855077335s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
1.533095088s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:05:30 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.533127529s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X..] duration:
3.461512159s trace:
https://jaeger.dev.storj.io/trace/6d35f8d6110e790f
FAIL (2025-12-08 16:05:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.461607891s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
8.713671334s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:05:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 8.713700179s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X.] duration:
3.55891942s trace:
https://jaeger.dev.storj.io/trace/6d35f8d6110e790f
FAIL (2025-12-08 16:05:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.55903948s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...X.........................X..X...X.................X.X...] duration:
5.984691433s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:05:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.984717141s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X.XX.XX..X.....XX...X.X.X.X....XX......X....X...............] duration:
5.832583816s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:04:57 UTC)
orbiter warning: context deadline exceeded
Duration: 5.832611197s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
1.609812382s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:04:53 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.609885993s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
5.937714513s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:04:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.937747775s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [X.........X...X..............X.............X................] duration:
7.469099568s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:04:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 7.469130707s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
1.513761512s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:04:30 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.513795154s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [....X.........................X..X...X.................X.X..] duration:
6.034298404s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:04:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.034327729s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX.XX.XX..X.....XX...X.X.X.X....XX......X....X..............] duration:
18.772946866s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:04:26 UTC)
orbiter warning: context deadline exceeded
Duration: 18.772973144s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X.........X...X..............X.............X...............] duration:
7.142193079s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:04:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 7.142220723s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.985736986s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:04:14 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.985766863s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.....X.........................X..X...X.................X.X.] duration:
5.864561532s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:04:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.864577433s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [......X.........................X..X...X.................X.X] duration:
1m0.001434219s trace:
https://jaeger.dev.storj.io/trace/690f79ef204fee2
FAIL (2025-12-08 16:03:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001458486s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [.X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X] duration:
5.001547929s trace:
https://jaeger.dev.storj.io/trace/6d35f8d6110e790f
FAIL (2025-12-08 16:03:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001577525s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.293156891s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:03:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.293190094s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X.............] duration:
16.472928346s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:03:45 UTC)
orbiter warning: context deadline exceeded
Duration: 16.472955497s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
6.149272038s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:03:32 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.149326082s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
1.524174758s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:03:32 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.52420196s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X.........X...X..............X.............X..............] duration:
6.238081812s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:03:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.238107791s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [..X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....] duration:
3.429829677s trace:
https://jaeger.dev.storj.io/trace/42949bd00e591a18
FAIL (2025-12-08 16:03:21 UTC)
orbiter warning: context deadline exceeded
Duration: 3.429861358s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
1.53751297s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:03:04 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.537540643s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X....] duration:
3.563113277s trace:
https://jaeger.dev.storj.io/trace/42949bd00e591a18
FAIL (2025-12-08 16:02:59 UTC)
orbiter warning: context deadline exceeded
Duration: 3.563153182s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.157916359s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:02:46 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.157944633s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
4.726260577s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 16:02:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.726278951s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X.XX.XX.XX..X.....XX...X.X.X.X....XX......X....X............] duration:
6.039573688s trace:
https://jaeger.dev.storj.io/trace/4f3b7f7689e9a093
FAIL (2025-12-08 16:02:33 UTC)
orbiter warning: context deadline exceeded
Duration: 6.039603404s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X.........X...X..............X.............X.............] duration:
6.099070658s trace:
https://jaeger.dev.storj.io/trace/4213b059398cd95c
FAIL (2025-12-08 16:02:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded
======LOGS======
Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload
Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload
Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments...
Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false
Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment...
Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment.
Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece...
Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment
Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments...
Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil>
Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold
Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments...
Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil>
Duration: 6.099110483s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X...] duration:
3.35751774s trace:
https://jaeger.dev.storj.io/trace/42949bd00e591a18
FAIL (2025-12-08 16:02:27 UTC)
orbiter warning: context deadline exceeded
Duration: 3.357547827s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.089080799s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 16:02:18 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.089109412s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
1.585438637s trace:
https://jaeger.dev.storj.io/trace/3bdb8560e64b821d
FAIL (2025-12-08 16:02:07 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.585468043s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......X.........................X..X...X.................X.] duration:
5.857997692s trace:
https://jaeger.dev.storj.io/trace/59a6790b91233307
FAIL (2025-12-08 16:02:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.858025245s
core
uplink
us-select-1
us1-placement-20
|