Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [....X....X....X......XX......X......X..X.....XXXX.X....XXX..] duration:
3.520328849s trace:
https://jaeger.dev.storj.io/trace/76d46c314fab7319
FAIL (2025-12-25 14:55:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.520361141s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................................XX....X.....X...X] duration:
30.002984392s trace:
https://jaeger.dev.storj.io/trace/7e027e932241536e
FAIL (2025-12-25 14:55:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA) context deadline exceeded
======LOGS======
c 25 14:54:41.386219623 (479h25m41.257302658s) [upload=420831]: Starting upload
Dec 25 14:54:41.386279507 (479h25m41.257362552s) [upload=420831]: Done starting upload
Dec 25 14:54:41.386409464 (479h25m41.257492519s) [upload=420831]: Uploading segments...
Dec 25 14:54:41.386520575 (479h25m41.25760361s) [upload=420831]: Got next segment. Inline: false
Dec 25 14:54:41.386555011 (479h25m41.257638046s) [upload=420831]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:54:41.386559891 (479h25m41.257642926s) [upload=420831]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:54:41.633320544 (479h25m41.504403589s) [seg_pos={0 0}, upload=420831]: Begin upload segment...
Dec 25 14:54:41.633367013 (479h25m41.504450049s) [seg_pos={0 0}, upload=420831]: Done begin upload segment.
Dec 25 14:54:41.633413281 (479h25m41.504496326s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.633431296 (479h25m41.504514341s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.634025067 (479h25m41.505108112s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.634217192 (479h25m41.505300227s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.634351808 (479h25m41.505434843s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.634554925 (479h25m41.50563797s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.634637292 (479h25m41.505720327s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.634896476 (479h25m41.505979521s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.635238005 (479h25m41.506321041s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.635486950 (479h25m41.506569995s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.635810155 (479h25m41.506893191s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.636217401 (479h25m41.507300436s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.636449713 (479h25m41.507532758s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.636912785 (479h25m41.50799582s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.637274323 (479h25m41.508357369s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:41.637490735 (479h25m41.50857377s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Uploading piece...
Dec 25 14:54:43.344619646 (479h25m43.215702681s) [upload=420831]: Next returned nil segment
Dec 25 14:54:43.344633662 (479h25m43.215716698s) [upload=420831]: Waiting for error group managing segments...
Dec 25 14:54:43.635421121 (479h25m43.506504156s) [noise=true , address=145.79.227.19:4078 , piece_id=3XKHHDK2JAQHKJL6QJUXVNZL56X2GVBEXROUVLEAQYULLLBDZURA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.659355095 (479h25m43.530438131s) [noise=true , address=145.79.227.48:4301 , piece_id=DK54OW3OKVJ6KDH5EGLKZ2RLPSBXGJE45NGUNVEK5CYDCKQLQXUQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.661034643 (479h25m43.532117678s) [noise=true , address=145.79.227.50:4374 , piece_id=5HUHIY3NZY2ZPFAY53YIHVRK7DS4JQ3E26WH2AVVTU4BZ54BHUZA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.664948956 (479h25m43.536032002s) [noise=true , address=145.79.227.49:4337 , piece_id=QM7Y2ZUX6JC3I3XO7ERD3UAX2FRT2DVQRB3MBKE5CJJSRYXLMWFA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.667524771 (479h25m43.538607816s) [noise=true , address=145.79.227.52:4447 , piece_id=TPY7BFXJ34PCQH7P45DSHWTRCLRDFA3EKEFOFR2AHBHQCK3RASYQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.670061091 (479h25m43.541144136s) [noise=true , address=145.79.227.39:4015 , piece_id=DEMDDUAW25PQZWLOLWJECCPFAJCZV6LQFYXDCEIJFEY6MSTHNQBQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.670302521 (479h25m43.541385566s) [noise=true , address=145.79.227.42:4121 , piece_id=MRUUEI7E5MZMUH2FA5WQNRZDKW7SBY2SG5SPXAPKGNKRCVVANXLA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.687635275 (479h25m43.558718321s) [noise=true , address=145.79.227.37:4565 , piece_id=ANXBJYMZ6AIG2TXNF7QSF7TKZ232UOZWHM3RVQUPV52P7VS56GDQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.693682890 (479h25m43.564765935s) [noise=true , address=145.79.227.27:4690 , piece_id=MEBFFHCZRSC4E7KU4PEBFFFYS3NTPH5JT5PV4Q3N6TRGHNALOSOA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:43.877985584 (479h25m43.749068619s) [noise=true , address=45.91.116.8:4046 , piece_id=UMSHLBI3LOAND3XIARVHLWPH7GTHIZCJ2MW5R6HG3MHNJDSQRRRA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:44.008601690 (479h25m43.879684735s) [noise=true , address=45.91.116.13:4154 , piece_id=UAONWIL3UP7F4653AYGKULRUWJA3AHTUC4OTFZEK7RCIUHPKK3YA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:44.081131228 (479h25m43.952214273s) [noise=true , address=45.91.116.5:4300 , piece_id=XSCFI3CAWHXYFTWDE67T3XJNWSQILGXMRG2MDEER3LLU2ZOZ2RMA, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:44.564946163 (479h25m44.436029208s) [noise=true , address=77.90.31.14:4663 , piece_id=ZD2X5E4YGIIAULPEIYAF7MXXHFRWID7G2G3MHCL7WSJHQAGQ2E5A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: <nil>
Dec 25 14:54:44.564965319 (479h25m44.436048365s) [seg_pos={0 0}, upload=420831]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:54:44.565226065 (479h25m44.436309101s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->45.91.116.9:4261: use of closed network connection
Dec 25 14:54:44.565233340 (479h25m44.436316385s) [noise=true , address=45.91.116.9:4261 , piece_id=3FU6KJ5DCJOEKVHDCB7GWYWO6VS4EFL7F2V4OOENQFFHGFQ32PVQ, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold
Dec 25 14:54:44.565408654 (479h25m44.436491689s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 25 14:54:44.565423412 (479h25m44.436506447s) [noise=true , address=145.79.227.40:4049 , piece_id=4VGKMGERSBPRCRPYTQSE3C5AHR5QQXHGH7DHFKBVQZFUAJ43N76A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold
Dec 25 14:54:44.565545745 (479h25m44.43662882s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 25 14:54:44.565570361 (479h25m44.436653426s) [noise=true , address=145.79.227.34:4905 , piece_id=NJAEVQMRBTJNGEYYNMPRBBVLUB7JSFYZBYRSH2E2SNVR2LQ5ZG3A, seg_pos={0 0}, upload=420831]: Long tail cancellation due to optimal threshold
Dec 25 14:54:44.565598124 (479h25m44.436681169s) [seg_pos={0 0}, upload=420831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:54:44.565721950 (479h25m44.436804996s) [upload=420831]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:54:44.565728643 (479h25m44.436811678s) [upload=420831]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:54:44.852842896 (479h25m44.723925931s) [upload=420831]: Done uploading segments...
Dec 25 14:54:44.852861713 (479h25m44.723944748s) [upload=420831]: Upload finished. err: <nil>
Duration: 30.003006985s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X....X.....] duration:
52.768517718s trace:
https://jaeger.dev.storj.io/trace/648a4d9ea8398e98
FAIL (2025-12-25 14:54:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded
======LOGS======
LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment
Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments...
Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF
Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection
Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments...
Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil>
Duration: 52.768544599s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [.....X....X....X......XX......X......X..X.....XXXX.X....XXX.] duration:
4.992318384s trace:
https://jaeger.dev.storj.io/trace/76d46c314fab7319
FAIL (2025-12-25 14:54:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.992352028s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
6.052967136s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:54:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.05299488s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
14.880331247s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:53:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.880353819s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................................XX....X.....X...] duration:
5.735238097s trace:
https://jaeger.dev.storj.io/trace/5c6e6dfc051e52ef
FAIL (2025-12-25 14:53:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
segments...
Dec 25 14:50:12.840624345 (479h21m12.71170739s) [upload=420766]: Got next segment. Inline: false
Dec 25 14:50:12.840666064 (479h21m12.711749099s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:50:12.840673409 (479h21m12.711756454s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:50:13.090470263 (479h21m12.961553318s) [seg_pos={0 0}, upload=420766]: Begin upload segment...
Dec 25 14:50:13.090522483 (479h21m12.961605528s) [seg_pos={0 0}, upload=420766]: Done begin upload segment.
Dec 25 14:50:13.090556799 (479h21m12.961639844s) [noise=true , address=145.79.227.32:4835 , piece_id=RXFICRQ5ET2RCLWWMGIA5AL7CRB7UNSOSWJEKOGHIUJRAXWNAI3Q, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.090581696 (479h21m12.961664741s) [noise=true , address=45.91.116.11:4118 , piece_id=MSLVAIT7ECXBKVLDU3WTBRUDZMUMFOXWNUC53YQFDWPY3S67TRXA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.091199131 (479h21m12.962282176s) [noise=true , address=145.79.227.50:4374 , piece_id=R5MFXY3LJYKB75MZKTKEVOAXVL2KLE2SDLY5E7BBCPDMMKWHHRHQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.091804204 (479h21m12.962887239s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.092292394 (479h21m12.963375439s) [noise=true , address=77.90.31.10:4517 , piece_id=VUE5BNVWZSO3KVTAX7PZATXZPZH7OBELANZXGICYN4R4WQYGH4CA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.092738543 (479h21m12.963821588s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.093376888 (479h21m12.964459934s) [noise=true , address=45.91.116.14:4189 , piece_id=FLVVZQFS43CQ6DSXMZW5HEGJKODLOJ2WUSIIIZ2ASGI5MHORMKHA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.094577695 (479h21m12.96566074s) [noise=true , address=77.90.31.15:4700 , piece_id=PFJBBJM5OYVBJTDC2VTGV6LNGC4H6BKPU54XHCZWWDMJ5OJV35RQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.095039725 (479h21m12.96612277s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.095915252 (479h21m12.966998297s) [noise=true , address=145.79.227.35:4483 , piece_id=63KSJDZ76VDDWQXIMIK735OFQO345EEPH7BH7FACYVNRV7FABYPQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.097448882 (479h21m12.968531927s) [noise=true , address=145.79.227.43:4158 , piece_id=ZQK3Z5BHKVL6SKHFBVMD7MXIAMU4CPJE7B7E6YRUJHHL6U6Q7QOA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.097876095 (479h21m12.968959131s) [noise=true , address=145.79.227.47:4266 , piece_id=B7EFQ2NUIGTMKTG6MFKYYCMEGWBBY2VRAGSXZT4FAJ3K565BBYBA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.099607032 (479h21m12.970690077s) [noise=true , address=145.79.227.4:4187 , piece_id=UKORVVPFJGGYVVQEN43EE7HCOL5DJYUYVBN4YQYGTABYW7RLYSHA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.099746908 (479h21m12.970829943s) [noise=true , address=145.79.227.7:4294 , piece_id=AUXLERYSGL3OM54WYQQ2E43ULMNGEBTM5SW5SK4OW3LEAWCJKTIA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.101657777 (479h21m12.972740822s) [noise=true , address=145.79.227.15:4875 , piece_id=H5YCKWGNJO2XBL232MSY2ZE4AJ2H77NUTKDGJJ6ZW4FIN6UO2K4A, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.103170727 (479h21m12.974253762s) [noise=true , address=145.79.227.52:4447 , piece_id=ULANCBPJDAYIYQJ35GLX5LCT3GODYQGYHD7V47N3ZUNTZANIBETA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:14.711523562 (479h21m14.582606607s) [upload=420766]: Next returned nil segment
Dec 25 14:50:14.711548981 (479h21m14.582632026s) [upload=420766]: Waiting for error group managing segments...
Dec 25 14:50:15.099664357 (479h21m14.970747392s) [noise=true , address=145.79.227.50:4374 , piece_id=R5MFXY3LJYKB75MZKTKEVOAXVL2KLE2SDLY5E7BBCPDMMKWHHRHQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.109514183 (479h21m14.980597218s) [noise=true , address=145.79.227.43:4158 , piece_id=ZQK3Z5BHKVL6SKHFBVMD7MXIAMU4CPJE7B7E6YRUJHHL6U6Q7QOA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.112105036 (479h21m14.983188081s) [noise=true , address=145.79.227.4:4187 , piece_id=UKORVVPFJGGYVVQEN43EE7HCOL5DJYUYVBN4YQYGTABYW7RLYSHA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.113593370 (479h21m14.984676405s) [noise=true , address=145.79.227.52:4447 , piece_id=ULANCBPJDAYIYQJ35GLX5LCT3GODYQGYHD7V47N3ZUNTZANIBETA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.127355866 (479h21m14.998438901s) [noise=true , address=145.79.227.32:4835 , piece_id=RXFICRQ5ET2RCLWWMGIA5AL7CRB7UNSOSWJEKOGHIUJRAXWNAI3Q, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.128702831 (479h21m14.999785876s) [noise=true , address=145.79.227.15:4875 , piece_id=H5YCKWGNJO2XBL232MSY2ZE4AJ2H77NUTKDGJJ6ZW4FIN6UO2K4A, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.132783281 (479h21m15.003866316s) [noise=true , address=145.79.227.7:4294 , piece_id=AUXLERYSGL3OM54WYQQ2E43ULMNGEBTM5SW5SK4OW3LEAWCJKTIA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.134009957 (479h21m15.005092992s) [noise=true , address=145.79.227.35:4483 , piece_id=63KSJDZ76VDDWQXIMIK735OFQO345EEPH7BH7FACYVNRV7FABYPQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.142476729 (479h21m15.013559764s) [noise=true , address=145.79.227.47:4266 , piece_id=B7EFQ2NUIGTMKTG6MFKYYCMEGWBBY2VRAGSXZT4FAJ3K565BBYBA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.386720420 (479h21m15.257803465s) [noise=true , address=77.90.31.10:4517 , piece_id=VUE5BNVWZSO3KVTAX7PZATXZPZH7OBELANZXGICYN4R4WQYGH4CA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.388154711 (479h21m15.259237756s) [noise=true , address=77.90.31.15:4700 , piece_id=PFJBBJM5OYVBJTDC2VTGV6LNGC4H6BKPU54XHCZWWDMJ5OJV35RQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.473735963 (479h21m15.344818998s) [noise=true , address=45.91.116.14:4189 , piece_id=FLVVZQFS43CQ6DSXMZW5HEGJKODLOJ2WUSIIIZ2ASGI5MHORMKHA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.547806295 (479h21m15.41888934s) [noise=true , address=45.91.116.11:4118 , piece_id=MSLVAIT7ECXBKVLDU3WTBRUDZMUMFOXWNUC53YQFDWPY3S67TRXA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.547832996 (479h21m15.418916041s) [seg_pos={0 0}, upload=420766]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:50:15.548163705 (479h21m15.41924675s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection
Dec 25 14:50:15.548186790 (479h21m15.419269835s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548199824 (479h21m15.419282869s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection
Dec 25 14:50:15.548211016 (479h21m15.419294081s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548340602 (479h21m15.419423637s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 25 14:50:15.548361993 (479h21m15.419445038s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548379466 (479h21m15.419462501s) [seg_pos={0 0}, upload=420766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:50:15.548463186 (479h21m15.419546221s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:50:15.548474748 (479h21m15.419557783s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:50:15.855111837 (479h21m15.726194872s) [upload=420766]: Done uploading segments...
Dec 25 14:50:15.855135421 (479h21m15.726218456s) [upload=420766]: Upload finished. err: <nil>
Duration: 5.735263167s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X....X....] duration:
54.842917113s trace:
https://jaeger.dev.storj.io/trace/648a4d9ea8398e98
FAIL (2025-12-25 14:53:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded
======LOGS======
LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment
Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments...
Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF
Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection
Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments...
Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil>
Duration: 54.842940988s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [......X....X....X......XX......X......X..X.....XXXX.X....XXX] duration:
5.001600368s trace:
https://jaeger.dev.storj.io/trace/76d46c314fab7319
FAIL (2025-12-25 14:53:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001624335s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [.......X....X....X......XX......X......X..X.....XXXX.X....XX] duration:
5.000925844s trace:
https://jaeger.dev.storj.io/trace/376160ce71dd525f
FAIL (2025-12-25 14:53:09 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000951383s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
12.746143122s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:52:54 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.746169582s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [X.......X....X....X......XX......X......X..X.....XXXX.X....X] duration:
5.00156864s trace:
https://jaeger.dev.storj.io/trace/5ba392f006d9668e
FAIL (2025-12-25 14:52:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00166401s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X....X...] duration:
42.609509216s trace:
https://jaeger.dev.storj.io/trace/648a4d9ea8398e98
FAIL (2025-12-25 14:52:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded
======LOGS======
LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment
Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments...
Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF
Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection
Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments...
Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil>
Duration: 42.609526959s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................................XX....X.....X..] duration:
6.034019619s trace:
https://jaeger.dev.storj.io/trace/5c6e6dfc051e52ef
FAIL (2025-12-25 14:52:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
segments...
Dec 25 14:50:12.840624345 (479h21m12.71170739s) [upload=420766]: Got next segment. Inline: false
Dec 25 14:50:12.840666064 (479h21m12.711749099s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:50:12.840673409 (479h21m12.711756454s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:50:13.090470263 (479h21m12.961553318s) [seg_pos={0 0}, upload=420766]: Begin upload segment...
Dec 25 14:50:13.090522483 (479h21m12.961605528s) [seg_pos={0 0}, upload=420766]: Done begin upload segment.
Dec 25 14:50:13.090556799 (479h21m12.961639844s) [noise=true , address=145.79.227.32:4835 , piece_id=RXFICRQ5ET2RCLWWMGIA5AL7CRB7UNSOSWJEKOGHIUJRAXWNAI3Q, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.090581696 (479h21m12.961664741s) [noise=true , address=45.91.116.11:4118 , piece_id=MSLVAIT7ECXBKVLDU3WTBRUDZMUMFOXWNUC53YQFDWPY3S67TRXA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.091199131 (479h21m12.962282176s) [noise=true , address=145.79.227.50:4374 , piece_id=R5MFXY3LJYKB75MZKTKEVOAXVL2KLE2SDLY5E7BBCPDMMKWHHRHQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.091804204 (479h21m12.962887239s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.092292394 (479h21m12.963375439s) [noise=true , address=77.90.31.10:4517 , piece_id=VUE5BNVWZSO3KVTAX7PZATXZPZH7OBELANZXGICYN4R4WQYGH4CA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.092738543 (479h21m12.963821588s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.093376888 (479h21m12.964459934s) [noise=true , address=45.91.116.14:4189 , piece_id=FLVVZQFS43CQ6DSXMZW5HEGJKODLOJ2WUSIIIZ2ASGI5MHORMKHA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.094577695 (479h21m12.96566074s) [noise=true , address=77.90.31.15:4700 , piece_id=PFJBBJM5OYVBJTDC2VTGV6LNGC4H6BKPU54XHCZWWDMJ5OJV35RQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.095039725 (479h21m12.96612277s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.095915252 (479h21m12.966998297s) [noise=true , address=145.79.227.35:4483 , piece_id=63KSJDZ76VDDWQXIMIK735OFQO345EEPH7BH7FACYVNRV7FABYPQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.097448882 (479h21m12.968531927s) [noise=true , address=145.79.227.43:4158 , piece_id=ZQK3Z5BHKVL6SKHFBVMD7MXIAMU4CPJE7B7E6YRUJHHL6U6Q7QOA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.097876095 (479h21m12.968959131s) [noise=true , address=145.79.227.47:4266 , piece_id=B7EFQ2NUIGTMKTG6MFKYYCMEGWBBY2VRAGSXZT4FAJ3K565BBYBA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.099607032 (479h21m12.970690077s) [noise=true , address=145.79.227.4:4187 , piece_id=UKORVVPFJGGYVVQEN43EE7HCOL5DJYUYVBN4YQYGTABYW7RLYSHA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.099746908 (479h21m12.970829943s) [noise=true , address=145.79.227.7:4294 , piece_id=AUXLERYSGL3OM54WYQQ2E43ULMNGEBTM5SW5SK4OW3LEAWCJKTIA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.101657777 (479h21m12.972740822s) [noise=true , address=145.79.227.15:4875 , piece_id=H5YCKWGNJO2XBL232MSY2ZE4AJ2H77NUTKDGJJ6ZW4FIN6UO2K4A, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.103170727 (479h21m12.974253762s) [noise=true , address=145.79.227.52:4447 , piece_id=ULANCBPJDAYIYQJ35GLX5LCT3GODYQGYHD7V47N3ZUNTZANIBETA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:14.711523562 (479h21m14.582606607s) [upload=420766]: Next returned nil segment
Dec 25 14:50:14.711548981 (479h21m14.582632026s) [upload=420766]: Waiting for error group managing segments...
Dec 25 14:50:15.099664357 (479h21m14.970747392s) [noise=true , address=145.79.227.50:4374 , piece_id=R5MFXY3LJYKB75MZKTKEVOAXVL2KLE2SDLY5E7BBCPDMMKWHHRHQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.109514183 (479h21m14.980597218s) [noise=true , address=145.79.227.43:4158 , piece_id=ZQK3Z5BHKVL6SKHFBVMD7MXIAMU4CPJE7B7E6YRUJHHL6U6Q7QOA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.112105036 (479h21m14.983188081s) [noise=true , address=145.79.227.4:4187 , piece_id=UKORVVPFJGGYVVQEN43EE7HCOL5DJYUYVBN4YQYGTABYW7RLYSHA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.113593370 (479h21m14.984676405s) [noise=true , address=145.79.227.52:4447 , piece_id=ULANCBPJDAYIYQJ35GLX5LCT3GODYQGYHD7V47N3ZUNTZANIBETA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.127355866 (479h21m14.998438901s) [noise=true , address=145.79.227.32:4835 , piece_id=RXFICRQ5ET2RCLWWMGIA5AL7CRB7UNSOSWJEKOGHIUJRAXWNAI3Q, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.128702831 (479h21m14.999785876s) [noise=true , address=145.79.227.15:4875 , piece_id=H5YCKWGNJO2XBL232MSY2ZE4AJ2H77NUTKDGJJ6ZW4FIN6UO2K4A, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.132783281 (479h21m15.003866316s) [noise=true , address=145.79.227.7:4294 , piece_id=AUXLERYSGL3OM54WYQQ2E43ULMNGEBTM5SW5SK4OW3LEAWCJKTIA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.134009957 (479h21m15.005092992s) [noise=true , address=145.79.227.35:4483 , piece_id=63KSJDZ76VDDWQXIMIK735OFQO345EEPH7BH7FACYVNRV7FABYPQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.142476729 (479h21m15.013559764s) [noise=true , address=145.79.227.47:4266 , piece_id=B7EFQ2NUIGTMKTG6MFKYYCMEGWBBY2VRAGSXZT4FAJ3K565BBYBA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.386720420 (479h21m15.257803465s) [noise=true , address=77.90.31.10:4517 , piece_id=VUE5BNVWZSO3KVTAX7PZATXZPZH7OBELANZXGICYN4R4WQYGH4CA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.388154711 (479h21m15.259237756s) [noise=true , address=77.90.31.15:4700 , piece_id=PFJBBJM5OYVBJTDC2VTGV6LNGC4H6BKPU54XHCZWWDMJ5OJV35RQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.473735963 (479h21m15.344818998s) [noise=true , address=45.91.116.14:4189 , piece_id=FLVVZQFS43CQ6DSXMZW5HEGJKODLOJ2WUSIIIZ2ASGI5MHORMKHA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.547806295 (479h21m15.41888934s) [noise=true , address=45.91.116.11:4118 , piece_id=MSLVAIT7ECXBKVLDU3WTBRUDZMUMFOXWNUC53YQFDWPY3S67TRXA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.547832996 (479h21m15.418916041s) [seg_pos={0 0}, upload=420766]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:50:15.548163705 (479h21m15.41924675s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection
Dec 25 14:50:15.548186790 (479h21m15.419269835s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548199824 (479h21m15.419282869s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection
Dec 25 14:50:15.548211016 (479h21m15.419294081s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548340602 (479h21m15.419423637s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 25 14:50:15.548361993 (479h21m15.419445038s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548379466 (479h21m15.419462501s) [seg_pos={0 0}, upload=420766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:50:15.548463186 (479h21m15.419546221s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:50:15.548474748 (479h21m15.419557783s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:50:15.855111837 (479h21m15.726194872s) [upload=420766]: Done uploading segments...
Dec 25 14:50:15.855135421 (479h21m15.726218456s) [upload=420766]: Upload finished. err: <nil>
Duration: 6.034046371s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
12.701254065s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:52:27 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.701270817s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [XX.......X....X....X......XX......X......X..X.....XXXX.X....] duration:
3.564244701s trace:
https://jaeger.dev.storj.io/trace/3431afd1456f3db4
FAIL (2025-12-25 14:52:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.564272835s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
11.402491543s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:51:57 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.402520778s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X....X..] duration:
50.360741224s trace:
https://jaeger.dev.storj.io/trace/648a4d9ea8398e98
FAIL (2025-12-25 14:51:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded
======LOGS======
LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment
Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments...
Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF
Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection
Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments...
Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil>
Duration: 50.360772183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
11.001332026s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:51:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.001355882s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................................XX....X.....X.] duration:
5.746840082s trace:
https://jaeger.dev.storj.io/trace/5c6e6dfc051e52ef
FAIL (2025-12-25 14:51:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
segments...
Dec 25 14:50:12.840624345 (479h21m12.71170739s) [upload=420766]: Got next segment. Inline: false
Dec 25 14:50:12.840666064 (479h21m12.711749099s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:50:12.840673409 (479h21m12.711756454s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:50:13.090470263 (479h21m12.961553318s) [seg_pos={0 0}, upload=420766]: Begin upload segment...
Dec 25 14:50:13.090522483 (479h21m12.961605528s) [seg_pos={0 0}, upload=420766]: Done begin upload segment.
Dec 25 14:50:13.090556799 (479h21m12.961639844s) [noise=true , address=145.79.227.32:4835 , piece_id=RXFICRQ5ET2RCLWWMGIA5AL7CRB7UNSOSWJEKOGHIUJRAXWNAI3Q, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.090581696 (479h21m12.961664741s) [noise=true , address=45.91.116.11:4118 , piece_id=MSLVAIT7ECXBKVLDU3WTBRUDZMUMFOXWNUC53YQFDWPY3S67TRXA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.091199131 (479h21m12.962282176s) [noise=true , address=145.79.227.50:4374 , piece_id=R5MFXY3LJYKB75MZKTKEVOAXVL2KLE2SDLY5E7BBCPDMMKWHHRHQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.091804204 (479h21m12.962887239s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.092292394 (479h21m12.963375439s) [noise=true , address=77.90.31.10:4517 , piece_id=VUE5BNVWZSO3KVTAX7PZATXZPZH7OBELANZXGICYN4R4WQYGH4CA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.092738543 (479h21m12.963821588s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.093376888 (479h21m12.964459934s) [noise=true , address=45.91.116.14:4189 , piece_id=FLVVZQFS43CQ6DSXMZW5HEGJKODLOJ2WUSIIIZ2ASGI5MHORMKHA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.094577695 (479h21m12.96566074s) [noise=true , address=77.90.31.15:4700 , piece_id=PFJBBJM5OYVBJTDC2VTGV6LNGC4H6BKPU54XHCZWWDMJ5OJV35RQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.095039725 (479h21m12.96612277s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.095915252 (479h21m12.966998297s) [noise=true , address=145.79.227.35:4483 , piece_id=63KSJDZ76VDDWQXIMIK735OFQO345EEPH7BH7FACYVNRV7FABYPQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.097448882 (479h21m12.968531927s) [noise=true , address=145.79.227.43:4158 , piece_id=ZQK3Z5BHKVL6SKHFBVMD7MXIAMU4CPJE7B7E6YRUJHHL6U6Q7QOA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.097876095 (479h21m12.968959131s) [noise=true , address=145.79.227.47:4266 , piece_id=B7EFQ2NUIGTMKTG6MFKYYCMEGWBBY2VRAGSXZT4FAJ3K565BBYBA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.099607032 (479h21m12.970690077s) [noise=true , address=145.79.227.4:4187 , piece_id=UKORVVPFJGGYVVQEN43EE7HCOL5DJYUYVBN4YQYGTABYW7RLYSHA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.099746908 (479h21m12.970829943s) [noise=true , address=145.79.227.7:4294 , piece_id=AUXLERYSGL3OM54WYQQ2E43ULMNGEBTM5SW5SK4OW3LEAWCJKTIA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.101657777 (479h21m12.972740822s) [noise=true , address=145.79.227.15:4875 , piece_id=H5YCKWGNJO2XBL232MSY2ZE4AJ2H77NUTKDGJJ6ZW4FIN6UO2K4A, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.103170727 (479h21m12.974253762s) [noise=true , address=145.79.227.52:4447 , piece_id=ULANCBPJDAYIYQJ35GLX5LCT3GODYQGYHD7V47N3ZUNTZANIBETA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:14.711523562 (479h21m14.582606607s) [upload=420766]: Next returned nil segment
Dec 25 14:50:14.711548981 (479h21m14.582632026s) [upload=420766]: Waiting for error group managing segments...
Dec 25 14:50:15.099664357 (479h21m14.970747392s) [noise=true , address=145.79.227.50:4374 , piece_id=R5MFXY3LJYKB75MZKTKEVOAXVL2KLE2SDLY5E7BBCPDMMKWHHRHQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.109514183 (479h21m14.980597218s) [noise=true , address=145.79.227.43:4158 , piece_id=ZQK3Z5BHKVL6SKHFBVMD7MXIAMU4CPJE7B7E6YRUJHHL6U6Q7QOA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.112105036 (479h21m14.983188081s) [noise=true , address=145.79.227.4:4187 , piece_id=UKORVVPFJGGYVVQEN43EE7HCOL5DJYUYVBN4YQYGTABYW7RLYSHA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.113593370 (479h21m14.984676405s) [noise=true , address=145.79.227.52:4447 , piece_id=ULANCBPJDAYIYQJ35GLX5LCT3GODYQGYHD7V47N3ZUNTZANIBETA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.127355866 (479h21m14.998438901s) [noise=true , address=145.79.227.32:4835 , piece_id=RXFICRQ5ET2RCLWWMGIA5AL7CRB7UNSOSWJEKOGHIUJRAXWNAI3Q, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.128702831 (479h21m14.999785876s) [noise=true , address=145.79.227.15:4875 , piece_id=H5YCKWGNJO2XBL232MSY2ZE4AJ2H77NUTKDGJJ6ZW4FIN6UO2K4A, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.132783281 (479h21m15.003866316s) [noise=true , address=145.79.227.7:4294 , piece_id=AUXLERYSGL3OM54WYQQ2E43ULMNGEBTM5SW5SK4OW3LEAWCJKTIA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.134009957 (479h21m15.005092992s) [noise=true , address=145.79.227.35:4483 , piece_id=63KSJDZ76VDDWQXIMIK735OFQO345EEPH7BH7FACYVNRV7FABYPQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.142476729 (479h21m15.013559764s) [noise=true , address=145.79.227.47:4266 , piece_id=B7EFQ2NUIGTMKTG6MFKYYCMEGWBBY2VRAGSXZT4FAJ3K565BBYBA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.386720420 (479h21m15.257803465s) [noise=true , address=77.90.31.10:4517 , piece_id=VUE5BNVWZSO3KVTAX7PZATXZPZH7OBELANZXGICYN4R4WQYGH4CA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.388154711 (479h21m15.259237756s) [noise=true , address=77.90.31.15:4700 , piece_id=PFJBBJM5OYVBJTDC2VTGV6LNGC4H6BKPU54XHCZWWDMJ5OJV35RQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.473735963 (479h21m15.344818998s) [noise=true , address=45.91.116.14:4189 , piece_id=FLVVZQFS43CQ6DSXMZW5HEGJKODLOJ2WUSIIIZ2ASGI5MHORMKHA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.547806295 (479h21m15.41888934s) [noise=true , address=45.91.116.11:4118 , piece_id=MSLVAIT7ECXBKVLDU3WTBRUDZMUMFOXWNUC53YQFDWPY3S67TRXA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.547832996 (479h21m15.418916041s) [seg_pos={0 0}, upload=420766]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:50:15.548163705 (479h21m15.41924675s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection
Dec 25 14:50:15.548186790 (479h21m15.419269835s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548199824 (479h21m15.419282869s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection
Dec 25 14:50:15.548211016 (479h21m15.419294081s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548340602 (479h21m15.419423637s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 25 14:50:15.548361993 (479h21m15.419445038s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548379466 (479h21m15.419462501s) [seg_pos={0 0}, upload=420766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:50:15.548463186 (479h21m15.419546221s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:50:15.548474748 (479h21m15.419557783s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:50:15.855111837 (479h21m15.726194872s) [upload=420766]: Done uploading segments...
Dec 25 14:50:15.855135421 (479h21m15.726218456s) [upload=420766]: Upload finished. err: <nil>
Duration: 5.746867434s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [XXX.......X....X....X......XX......X......X..X.....XXXX.X...] duration:
5.001040272s trace:
https://jaeger.dev.storj.io/trace/3431afd1456f3db4
FAIL (2025-12-25 14:51:19 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001061953s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
10.745720391s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:51:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.745747271s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [XXXX.......X....X....X......XX......X......X..X.....XXXX.X..] duration:
5.001217879s trace:
https://jaeger.dev.storj.io/trace/3431afd1456f3db4
FAIL (2025-12-25 14:50:52 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001244201s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration:
49.503549966s trace:
https://jaeger.dev.storj.io/trace/648a4d9ea8398e98
FAIL (2025-12-25 14:50:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded
======LOGS======
LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment
Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments...
Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF
Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection
Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments...
Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil>
Duration: 49.50362509s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................................XX....X.....X] duration:
30.003082735s trace:
https://jaeger.dev.storj.io/trace/5c6e6dfc051e52ef
FAIL (2025-12-25 14:50:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
segments...
Dec 25 14:50:12.840624345 (479h21m12.71170739s) [upload=420766]: Got next segment. Inline: false
Dec 25 14:50:12.840666064 (479h21m12.711749099s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:50:12.840673409 (479h21m12.711756454s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:50:13.090470263 (479h21m12.961553318s) [seg_pos={0 0}, upload=420766]: Begin upload segment...
Dec 25 14:50:13.090522483 (479h21m12.961605528s) [seg_pos={0 0}, upload=420766]: Done begin upload segment.
Dec 25 14:50:13.090556799 (479h21m12.961639844s) [noise=true , address=145.79.227.32:4835 , piece_id=RXFICRQ5ET2RCLWWMGIA5AL7CRB7UNSOSWJEKOGHIUJRAXWNAI3Q, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.090581696 (479h21m12.961664741s) [noise=true , address=45.91.116.11:4118 , piece_id=MSLVAIT7ECXBKVLDU3WTBRUDZMUMFOXWNUC53YQFDWPY3S67TRXA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.091199131 (479h21m12.962282176s) [noise=true , address=145.79.227.50:4374 , piece_id=R5MFXY3LJYKB75MZKTKEVOAXVL2KLE2SDLY5E7BBCPDMMKWHHRHQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.091804204 (479h21m12.962887239s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.092292394 (479h21m12.963375439s) [noise=true , address=77.90.31.10:4517 , piece_id=VUE5BNVWZSO3KVTAX7PZATXZPZH7OBELANZXGICYN4R4WQYGH4CA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.092738543 (479h21m12.963821588s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.093376888 (479h21m12.964459934s) [noise=true , address=45.91.116.14:4189 , piece_id=FLVVZQFS43CQ6DSXMZW5HEGJKODLOJ2WUSIIIZ2ASGI5MHORMKHA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.094577695 (479h21m12.96566074s) [noise=true , address=77.90.31.15:4700 , piece_id=PFJBBJM5OYVBJTDC2VTGV6LNGC4H6BKPU54XHCZWWDMJ5OJV35RQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.095039725 (479h21m12.96612277s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.095915252 (479h21m12.966998297s) [noise=true , address=145.79.227.35:4483 , piece_id=63KSJDZ76VDDWQXIMIK735OFQO345EEPH7BH7FACYVNRV7FABYPQ, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.097448882 (479h21m12.968531927s) [noise=true , address=145.79.227.43:4158 , piece_id=ZQK3Z5BHKVL6SKHFBVMD7MXIAMU4CPJE7B7E6YRUJHHL6U6Q7QOA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.097876095 (479h21m12.968959131s) [noise=true , address=145.79.227.47:4266 , piece_id=B7EFQ2NUIGTMKTG6MFKYYCMEGWBBY2VRAGSXZT4FAJ3K565BBYBA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.099607032 (479h21m12.970690077s) [noise=true , address=145.79.227.4:4187 , piece_id=UKORVVPFJGGYVVQEN43EE7HCOL5DJYUYVBN4YQYGTABYW7RLYSHA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.099746908 (479h21m12.970829943s) [noise=true , address=145.79.227.7:4294 , piece_id=AUXLERYSGL3OM54WYQQ2E43ULMNGEBTM5SW5SK4OW3LEAWCJKTIA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.101657777 (479h21m12.972740822s) [noise=true , address=145.79.227.15:4875 , piece_id=H5YCKWGNJO2XBL232MSY2ZE4AJ2H77NUTKDGJJ6ZW4FIN6UO2K4A, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:13.103170727 (479h21m12.974253762s) [noise=true , address=145.79.227.52:4447 , piece_id=ULANCBPJDAYIYQJ35GLX5LCT3GODYQGYHD7V47N3ZUNTZANIBETA, seg_pos={0 0}, upload=420766]: Uploading piece...
Dec 25 14:50:14.711523562 (479h21m14.582606607s) [upload=420766]: Next returned nil segment
Dec 25 14:50:14.711548981 (479h21m14.582632026s) [upload=420766]: Waiting for error group managing segments...
Dec 25 14:50:15.099664357 (479h21m14.970747392s) [noise=true , address=145.79.227.50:4374 , piece_id=R5MFXY3LJYKB75MZKTKEVOAXVL2KLE2SDLY5E7BBCPDMMKWHHRHQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.109514183 (479h21m14.980597218s) [noise=true , address=145.79.227.43:4158 , piece_id=ZQK3Z5BHKVL6SKHFBVMD7MXIAMU4CPJE7B7E6YRUJHHL6U6Q7QOA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.112105036 (479h21m14.983188081s) [noise=true , address=145.79.227.4:4187 , piece_id=UKORVVPFJGGYVVQEN43EE7HCOL5DJYUYVBN4YQYGTABYW7RLYSHA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.113593370 (479h21m14.984676405s) [noise=true , address=145.79.227.52:4447 , piece_id=ULANCBPJDAYIYQJ35GLX5LCT3GODYQGYHD7V47N3ZUNTZANIBETA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.127355866 (479h21m14.998438901s) [noise=true , address=145.79.227.32:4835 , piece_id=RXFICRQ5ET2RCLWWMGIA5AL7CRB7UNSOSWJEKOGHIUJRAXWNAI3Q, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.128702831 (479h21m14.999785876s) [noise=true , address=145.79.227.15:4875 , piece_id=H5YCKWGNJO2XBL232MSY2ZE4AJ2H77NUTKDGJJ6ZW4FIN6UO2K4A, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.132783281 (479h21m15.003866316s) [noise=true , address=145.79.227.7:4294 , piece_id=AUXLERYSGL3OM54WYQQ2E43ULMNGEBTM5SW5SK4OW3LEAWCJKTIA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.134009957 (479h21m15.005092992s) [noise=true , address=145.79.227.35:4483 , piece_id=63KSJDZ76VDDWQXIMIK735OFQO345EEPH7BH7FACYVNRV7FABYPQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.142476729 (479h21m15.013559764s) [noise=true , address=145.79.227.47:4266 , piece_id=B7EFQ2NUIGTMKTG6MFKYYCMEGWBBY2VRAGSXZT4FAJ3K565BBYBA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.386720420 (479h21m15.257803465s) [noise=true , address=77.90.31.10:4517 , piece_id=VUE5BNVWZSO3KVTAX7PZATXZPZH7OBELANZXGICYN4R4WQYGH4CA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.388154711 (479h21m15.259237756s) [noise=true , address=77.90.31.15:4700 , piece_id=PFJBBJM5OYVBJTDC2VTGV6LNGC4H6BKPU54XHCZWWDMJ5OJV35RQ, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.473735963 (479h21m15.344818998s) [noise=true , address=45.91.116.14:4189 , piece_id=FLVVZQFS43CQ6DSXMZW5HEGJKODLOJ2WUSIIIZ2ASGI5MHORMKHA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.547806295 (479h21m15.41888934s) [noise=true , address=45.91.116.11:4118 , piece_id=MSLVAIT7ECXBKVLDU3WTBRUDZMUMFOXWNUC53YQFDWPY3S67TRXA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: <nil>
Dec 25 14:50:15.547832996 (479h21m15.418916041s) [seg_pos={0 0}, upload=420766]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:50:15.548163705 (479h21m15.41924675s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38822->45.91.116.4:4940: use of closed network connection
Dec 25 14:50:15.548186790 (479h21m15.419269835s) [noise=true , address=45.91.116.4:4940 , piece_id=HJMEUG4WNM46S7KUDPKD44ZH6EPETQA5PTPGIOGQYK7EM5BY2FBA, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548199824 (479h21m15.419282869s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56660->45.91.116.8:4047: use of closed network connection
Dec 25 14:50:15.548211016 (479h21m15.419294081s) [noise=true , address=45.91.116.8:4047 , piece_id=GXNJ44OAYI4YGYUOXMDU6SI67LAPCRQERKI35WINFYKAVACMEX2Q, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548340602 (479h21m15.419423637s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 25 14:50:15.548361993 (479h21m15.419445038s) [noise=true , address=145.79.227.44:4193 , piece_id=I7T6PUBNGLISE3CKQKJ4KIP2HDMBAZBJ5GW5UJYANTR5O7O32HEA, seg_pos={0 0}, upload=420766]: Long tail cancellation due to optimal threshold
Dec 25 14:50:15.548379466 (479h21m15.419462501s) [seg_pos={0 0}, upload=420766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:50:15.548463186 (479h21m15.419546221s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:50:15.548474748 (479h21m15.419557783s) [upload=420766]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:50:15.855111837 (479h21m15.726194872s) [upload=420766]: Done uploading segments...
Dec 25 14:50:15.855135421 (479h21m15.726218456s) [upload=420766]: Upload finished. err: <nil>
Duration: 30.003108052s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
12.007565646s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:50:20 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.007594962s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
18 out of 60 is failed (threshold=6) [XXXXX.......X....X....X......XX......X......X..X.....XXXX.X.] duration:
5.001255004s trace:
https://jaeger.dev.storj.io/trace/3431afd1456f3db4
FAIL (2025-12-25 14:49:54 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001280131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X....X] duration:
1m0.00212371s trace:
https://jaeger.dev.storj.io/trace/648a4d9ea8398e98
FAIL (2025-12-25 14:49:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, Piece ID: D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA) context deadline exceeded; piecestore: (Node ID: 12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, Piece ID: DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q) context deadline exceeded; piecestore: (Node ID: 1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, Piece ID: CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ) context deadline exceeded
======LOGS======
LRFAJ4A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.219751443 (479h19m47.090834488s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220216107 (479h19m47.091299152s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.220712403 (479h19m47.091795438s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221271237 (479h19m47.092354282s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.221781458 (479h19m47.092864493s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222304945 (479h19m47.09338799s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.222860563 (479h19m47.093943598s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.223977880 (479h19m47.095060915s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.224588283 (479h19m47.095671328s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.225348952 (479h19m47.096431997s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.226650600 (479h19m47.097733635s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.227991203 (479h19m47.099074238s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.230531941 (479h19m47.101614986s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.231669336 (479h19m47.102752371s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.233153132 (479h19m47.104236177s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:47.234404344 (479h19m47.10548739s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Uploading piece...
Dec 25 14:48:48.806464342 (479h19m48.677547377s) [upload=420750]: Next returned nil segment
Dec 25 14:48:48.806477727 (479h19m48.677560762s) [upload=420750]: Waiting for error group managing segments...
Dec 25 14:48:49.210757804 (479h19m49.08184085s) [noise=true , address=145.79.227.25:4634 , piece_id=KADGCKV3JB5EH7UA3FAHK6G7E7CTQG7GJWXZJVEU6BO6775YWZWA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.213995910 (479h19m49.085078945s) [noise=true , address=145.79.227.11:4742 , piece_id=PUP7DAYP564VIDWUS7KFIYI4WJESCPYN3EYYYAVMPNMXZCGUQNPQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.256199994 (479h19m49.127283049s) [noise=true , address=145.79.227.24:4616 , piece_id=W5JMXESIZWRUFQK374KWKEU77BUYDJ7SSITX5DFAXQGCDJ5Y7NCA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.269370613 (479h19m49.140453648s) [noise=true , address=145.79.227.12:4786 , piece_id=WKZCNQ4ZGSEM2LLPA3KYZVJCVWT7XH5BKXJEHEGPUWZKTHJWOHLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.354549388 (479h19m49.225632433s) [noise=true , address=145.79.227.33:4901 , piece_id=L6PCUR4JYD6AN5HUQSZ6K3WTKTNZ3JQGCKFWIEEN6UONAGNPI6XA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/e6/log-00000000000124e6-00000000: read-only file system; EOF
Dec 25 14:48:49.361521644 (479h19m49.232604679s) [noise=true , address=77.90.31.19:4875 , piece_id=KPVFR5KM6XFURPVZ253J4IHNNCLRZVSPSWD6AXPGZ6LNHHU533EA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.460798523 (479h19m49.331881568s) [noise=true , address=77.90.31.10:4531 , piece_id=JWWLINUAM6MH54DXQRGXRP73RBKOW6C7KG6YUF6WO7XYXZG7QYBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.530744472 (479h19m49.401827507s) [noise=true , address=77.90.31.18:4828 , piece_id=47Q7GUF75NN4DNCJO2YCIVLGBQEFDQEIBO2AXIPZMUDW6UNVHLTA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:49.641219355 (479h19m49.51230239s) [noise=true , address=45.91.116.10:4092 , piece_id=KNR6CUIELLYFNH7XM2KRXCKN4TBBKYPDE37O342X7LU2ALRFAJ4A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.083235648 (479h19m49.954318693s) [noise=true , address=45.91.116.13:4157 , piece_id=CZUCGWQXWJHNTPBCPGKIVSW2F42QPYCN5WKHR4SMWIAWKD24XTLQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.125289847 (479h19m49.996372882s) [noise=true , address=145.79.227.30:4781 , piece_id=WBV3SDIOYGNBB5UCHS5L35HYX7C3I5UNBGGWL437SBMPZLEHRZ3A, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.151297690 (479h19m50.022380735s) [noise=true , address=145.79.227.40:4082 , piece_id=4KFQLMQ2PQ3IDU5FIUD6F5VPF2TYE4JPBVON5I4XXOCBGYPM47CA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.374739821 (479h19m50.245822856s) [noise=true , address=77.90.31.11:4559 , piece_id=B7EVCRCHMQV2ENDQUFJO7OPCTZ76ESYJFIKJOBDL33NUPDPDQTFQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.386210192 (479h19m50.257293227s) [noise=true , address=77.90.31.12:4620 , piece_id=DXKODMFRE4M5D6T5FT65HM7HENLFWQVPC3KFP2GB6UZWPH6ADE3Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528905399 (479h19m50.399988444s) [noise=true , address=77.90.31.14:4681 , piece_id=D2MA3JKXQM6NTQAPOPFRPQSUPBTJKKX6X6CFWWGHLN2VTPI76FRA, seg_pos={0 0}, upload=420750]: Done uploading piece. err: <nil>
Dec 25 14:48:50.528930115 (479h19m50.40001316s) [seg_pos={0 0}, upload=420750]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:48:50.529248502 (479h19m50.400331547s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529258100 (479h19m50.400341145s) [noise=true , address=77.90.31.15:4731 , piece_id=VA3WJZPQU3PRO5NOIOHW4D3TL2Q7BAJ25AMF72IY3CEP6PMCFUWQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529375083 (479h19m50.400458138s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53542->64.49.13.169:4710: use of closed network connection
Dec 25 14:48:50.529387437 (479h19m50.400470492s) [noise=true , address=64.49.13.169:4710 , piece_id=HJDIQ23GO55XAHXPDE6XCV3H5EIXKZRCMFZVDIWRDINOO6RXL34Q, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529590733 (479h19m50.400673778s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF
Dec 25 14:48:50.529600192 (479h19m50.400683237s) [noise=true , address=145.79.227.44:4199 , piece_id=TTTATIFATKTFZCWLEBFZB2HEHLBYL23VBOBXKVBWTJZIZYPBVOBQ, seg_pos={0 0}, upload=420750]: Long tail cancellation due to optimal threshold
Dec 25 14:48:50.529622103 (479h19m50.400705148s) [seg_pos={0 0}, upload=420750]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:48:50.529741852 (479h19m50.400824887s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:48:50.529753624 (479h19m50.400836669s) [upload=420750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:48:50.822360597 (479h19m50.693443632s) [upload=420750]: Done uploading segments...
Dec 25 14:48:50.822379935 (479h19m50.69346298s) [upload=420750]: Upload finished. err: <nil>
Duration: 1m0.002132988s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................................XX....X.....] duration:
5.812741479s trace:
https://jaeger.dev.storj.io/trace/6c4af9bed16efc90
FAIL (2025-12-25 14:49:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
egments...
Dec 25 14:46:13.264075569 (479h17m13.135158614s) [upload=420712]: Got next segment. Inline: false
Dec 25 14:46:13.264162195 (479h17m13.13524524s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:46:13.264170851 (479h17m13.135253886s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:46:13.512525890 (479h17m13.383608946s) [seg_pos={0 0}, upload=420712]: Begin upload segment...
Dec 25 14:46:13.512581887 (479h17m13.383664943s) [seg_pos={0 0}, upload=420712]: Done begin upload segment.
Dec 25 14:46:13.512618988 (479h17m13.383702063s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.512636512 (479h17m13.383719547s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.513623511 (479h17m13.384706556s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514215328 (479h17m13.385298373s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514707735 (479h17m13.38579078s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515215002 (479h17m13.386298047s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515714012 (479h17m13.386797057s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516240404 (479h17m13.387323449s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516782386 (479h17m13.387865441s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517289491 (479h17m13.388372536s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517780436 (479h17m13.388863501s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517871841 (479h17m13.388954916s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518404334 (479h17m13.389487379s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518876013 (479h17m13.389959058s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519390463 (479h17m13.390473508s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519862140 (479h17m13.390945185s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:15.229763881 (479h17m15.100846916s) [upload=420712]: Next returned nil segment
Dec 25 14:46:15.229870384 (479h17m15.100953429s) [upload=420712]: Waiting for error group managing segments...
Dec 25 14:46:15.513913870 (479h17m15.384996905s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.525529769 (479h17m15.396612814s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.535294673 (479h17m15.406377718s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536111188 (479h17m15.407194233s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536510157 (479h17m15.407593203s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.537886388 (479h17m15.408969434s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.551463891 (479h17m15.422546936s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.557990359 (479h17m15.429073404s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.558774191 (479h17m15.429857226s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.609636352 (479h17m15.480719397s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.747129702 (479h17m15.618212747s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.800198454 (479h17m15.671281499s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853951790 (479h17m15.725034825s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853973741 (479h17m15.725056776s) [seg_pos={0 0}, upload=420712]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:46:15.854155878 (479h17m15.725238923s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection
Dec 25 14:46:15.854164465 (479h17m15.72524751s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854293810 (479h17m15.725376856s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 25 14:46:15.854306345 (479h17m15.72538938s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854356590 (479h17m15.725439625s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection
Dec 25 14:46:15.854363533 (479h17m15.725446568s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854377661 (479h17m15.725460706s) [seg_pos={0 0}, upload=420712]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:46:15.854448205 (479h17m15.72553134s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:46:15.854452914 (479h17m15.725535959s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:46:16.148710570 (479h17m16.019793605s) [upload=420712]: Done uploading segments...
Dec 25 14:46:16.148734706 (479h17m16.019817752s) [upload=420712]: Upload finished. err: <nil>
Duration: 5.812762097s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [XXXXXX.......X....X....X......XX......X......X..X.....XXXX.X] duration:
5.001278154s trace:
https://jaeger.dev.storj.io/trace/3431afd1456f3db4
FAIL (2025-12-25 14:49:32 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001300075s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................................XX....X....] duration:
6.199133194s trace:
https://jaeger.dev.storj.io/trace/6c4af9bed16efc90
FAIL (2025-12-25 14:49:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
egments...
Dec 25 14:46:13.264075569 (479h17m13.135158614s) [upload=420712]: Got next segment. Inline: false
Dec 25 14:46:13.264162195 (479h17m13.13524524s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:46:13.264170851 (479h17m13.135253886s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:46:13.512525890 (479h17m13.383608946s) [seg_pos={0 0}, upload=420712]: Begin upload segment...
Dec 25 14:46:13.512581887 (479h17m13.383664943s) [seg_pos={0 0}, upload=420712]: Done begin upload segment.
Dec 25 14:46:13.512618988 (479h17m13.383702063s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.512636512 (479h17m13.383719547s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.513623511 (479h17m13.384706556s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514215328 (479h17m13.385298373s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514707735 (479h17m13.38579078s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515215002 (479h17m13.386298047s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515714012 (479h17m13.386797057s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516240404 (479h17m13.387323449s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516782386 (479h17m13.387865441s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517289491 (479h17m13.388372536s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517780436 (479h17m13.388863501s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517871841 (479h17m13.388954916s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518404334 (479h17m13.389487379s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518876013 (479h17m13.389959058s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519390463 (479h17m13.390473508s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519862140 (479h17m13.390945185s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:15.229763881 (479h17m15.100846916s) [upload=420712]: Next returned nil segment
Dec 25 14:46:15.229870384 (479h17m15.100953429s) [upload=420712]: Waiting for error group managing segments...
Dec 25 14:46:15.513913870 (479h17m15.384996905s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.525529769 (479h17m15.396612814s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.535294673 (479h17m15.406377718s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536111188 (479h17m15.407194233s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536510157 (479h17m15.407593203s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.537886388 (479h17m15.408969434s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.551463891 (479h17m15.422546936s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.557990359 (479h17m15.429073404s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.558774191 (479h17m15.429857226s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.609636352 (479h17m15.480719397s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.747129702 (479h17m15.618212747s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.800198454 (479h17m15.671281499s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853951790 (479h17m15.725034825s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853973741 (479h17m15.725056776s) [seg_pos={0 0}, upload=420712]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:46:15.854155878 (479h17m15.725238923s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection
Dec 25 14:46:15.854164465 (479h17m15.72524751s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854293810 (479h17m15.725376856s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 25 14:46:15.854306345 (479h17m15.72538938s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854356590 (479h17m15.725439625s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection
Dec 25 14:46:15.854363533 (479h17m15.725446568s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854377661 (479h17m15.725460706s) [seg_pos={0 0}, upload=420712]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:46:15.854448205 (479h17m15.72553134s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:46:15.854452914 (479h17m15.725535959s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:46:16.148710570 (479h17m16.019793605s) [upload=420712]: Done uploading segments...
Dec 25 14:46:16.148734706 (479h17m16.019817752s) [upload=420712]: Upload finished. err: <nil>
Duration: 6.199161788s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
14.193181094s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:49:09 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.193206232s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
54.222095343s trace:
https://jaeger.dev.storj.io/trace/6fe4241072c742e3
FAIL (2025-12-25 14:48:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.48:4305 , piece_id=OG4GALOR3LJX2D224H7ZU7ZNMWFZA2DOYUDK75PK7RIJJZNC3ORQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292399703 (479h15m14.163482759s) [noise=true , address=64.49.13.140:4510 , piece_id=QMZV6CYJ3XFRXGD7TXNUNW56MZMVC52QTVEQAP7FNGD5GYBSVL3A, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292475207 (479h15m14.163558242s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292983274 (479h15m14.164066319s) [noise=true , address=145.79.227.35:4515 , piece_id=234QBCMNJQNPWR5SCI37NIMWD5XDUUYGKVNG6LA2DFOEV53WZZXA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293076572 (479h15m14.164159617s) [noise=true , address=145.79.227.30:4776 , piece_id=3BXXLR3PGZE53TQBQMHJT4O7UVFGCZNQALAFPCGEW3KLYSI2PMFA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293469279 (479h15m14.164552324s) [noise=true , address=45.91.116.6:4003 , piece_id=WFKJKH56B6JB2IAQRZJAEXOJAUYFD2YTFKNSZM6UYWCYOXMO72YA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293653110 (479h15m14.164736155s) [noise=true , address=145.79.227.3:4178 , piece_id=ISPXIDTSKSSG2R6CXR6YSXKUEGOCMG4ADIBXZZM6PJMPNLHMIWLA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294018936 (479h15m14.165101971s) [noise=true , address=77.90.31.17:4801 , piece_id=ABQWE7YGCVPYL6WD3TQQ7IYCLFUBTHK3G3LNQRHJRCDQN2PNXBAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294278030 (479h15m14.165361075s) [noise=true , address=45.91.116.12:4241 , piece_id=SXQDJRNWL3MNKKSMGR2SALBGUT5ES7EGD5HIY2NSGATTFBZTDLAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294534178 (479h15m14.165617223s) [noise=true , address=145.79.227.44:4225 , piece_id=LSBRYB2DMJCHZKBSGBP26FNM55VKRBZXFRCYC77CSYTU2JZ7PT7Q, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294981219 (479h15m14.166064264s) [noise=true , address=77.90.31.15:4723 , piece_id=NKKKWYITQV2SEZNZHPZFVY4DU4G24ZGPYUTWPTRX7U7K27GSGBAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.295163165 (479h15m14.16624621s) [noise=true , address=77.90.31.18:4836 , piece_id=7XYLKDNPRABN3BS3V375CXCT4NEWUW7RF3NRVVPTO4YAJFFOIPQQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.295595989 (479h15m14.166679034s) [noise=true , address=77.90.31.12:4610 , piece_id=6CK7RGQKSVTL5PB6Q2NRWCCCRYNMRFDUWXNWWRHXJ4WH523Y55DQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.296289399 (479h15m14.167372444s) [noise=true , address=45.91.116.11:4123 , piece_id=NKCCXXNYBKFWIREAP7GUQKYQQHFF7M5NDXO3XUSHYR2S4QJVBUNA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.296903900 (479h15m14.167986945s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.297479797 (479h15m14.168562842s) [noise=true , address=77.90.31.13:4654 , piece_id=WQ5KVXNI73SQWFXBQWTZ2JQFN2Y7KNYIUZZEI5RCFW3YLKQLAQGA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:15.616863706 (479h15m15.487946751s) [upload=420682]: Next returned nil segment
Dec 25 14:44:15.616879916 (479h15m15.487962951s) [upload=420682]: Waiting for error group managing segments...
Dec 25 14:44:15.950058414 (479h15m15.821141459s) [noise=true , address=64.49.13.140:4510 , piece_id=QMZV6CYJ3XFRXGD7TXNUNW56MZMVC52QTVEQAP7FNGD5GYBSVL3A, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.187623839 (479h15m16.058706884s) [noise=true , address=145.79.227.48:4305 , piece_id=OG4GALOR3LJX2D224H7ZU7ZNMWFZA2DOYUDK75PK7RIJJZNC3ORQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.195138668 (479h15m16.066221703s) [noise=true , address=145.79.227.3:4178 , piece_id=ISPXIDTSKSSG2R6CXR6YSXKUEGOCMG4ADIBXZZM6PJMPNLHMIWLA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.350781588 (479h15m16.221864623s) [noise=true , address=77.90.31.12:4610 , piece_id=6CK7RGQKSVTL5PB6Q2NRWCCCRYNMRFDUWXNWWRHXJ4WH523Y55DQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.396653487 (479h15m16.267736522s) [noise=true , address=45.91.116.11:4123 , piece_id=NKCCXXNYBKFWIREAP7GUQKYQQHFF7M5NDXO3XUSHYR2S4QJVBUNA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.431748113 (479h15m16.302831158s) [noise=true , address=45.91.116.6:4003 , piece_id=WFKJKH56B6JB2IAQRZJAEXOJAUYFD2YTFKNSZM6UYWCYOXMO72YA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.432579556 (479h15m16.303662591s) [noise=true , address=77.90.31.13:4654 , piece_id=WQ5KVXNI73SQWFXBQWTZ2JQFN2Y7KNYIUZZEI5RCFW3YLKQLAQGA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.463009770 (479h15m16.334092815s) [noise=true , address=77.90.31.17:4801 , piece_id=ABQWE7YGCVPYL6WD3TQQ7IYCLFUBTHK3G3LNQRHJRCDQN2PNXBAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.560546326 (479h15m16.431629361s) [noise=true , address=145.79.227.35:4515 , piece_id=234QBCMNJQNPWR5SCI37NIMWD5XDUUYGKVNG6LA2DFOEV53WZZXA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.611247740 (479h15m16.482330786s) [noise=true , address=77.90.31.15:4723 , piece_id=NKKKWYITQV2SEZNZHPZFVY4DU4G24ZGPYUTWPTRX7U7K27GSGBAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.758818880 (479h15m16.629901925s) [noise=true , address=77.90.31.18:4836 , piece_id=7XYLKDNPRABN3BS3V375CXCT4NEWUW7RF3NRVVPTO4YAJFFOIPQQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.782424619 (479h15m16.653507665s) [noise=true , address=45.91.116.12:4241 , piece_id=SXQDJRNWL3MNKKSMGR2SALBGUT5ES7EGD5HIY2NSGATTFBZTDLAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.072910743 (479h15m16.943993789s) [noise=true , address=145.79.227.30:4776 , piece_id=3BXXLR3PGZE53TQBQMHJT4O7UVFGCZNQALAFPCGEW3KLYSI2PMFA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.146569922 (479h15m17.017652967s) [noise=true , address=145.79.227.44:4225 , piece_id=LSBRYB2DMJCHZKBSGBP26FNM55VKRBZXFRCYC77CSYTU2JZ7PT7Q, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.146599148 (479h15m17.017682183s) [seg_pos={0 0}, upload=420682]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:44:17.146770013 (479h15m17.017853059s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.146776546 (479h15m17.017859581s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147100592 (479h15m17.018183638s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection
Dec 25 14:44:17.147108839 (479h15m17.018191884s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147251260 (479h15m17.018334305s) [noise=true , address=145.79.227.34:4915 , piece_id=GR66BPUQOFREGS2ZE36OB34SWX22AXIUASSDPIHR45ITW2F7ODCA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.147260467 (479h15m17.018343512s) [noise=true , address=145.79.227.34:4915 , piece_id=GR66BPUQOFREGS2ZE36OB34SWX22AXIUASSDPIHR45ITW2F7ODCA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147397388 (479h15m17.018480423s) [noise=true , address=77.90.31.11:4574 , piece_id=IJPN6SEXZWUEDR5DNBKP4RG3CJSXCZ7EEBSUSVQAGIEA54PAAPRA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.147405924 (479h15m17.018488969s) [noise=true , address=77.90.31.11:4574 , piece_id=IJPN6SEXZWUEDR5DNBKP4RG3CJSXCZ7EEBSUSVQAGIEA54PAAPRA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147425211 (479h15m17.018508256s) [seg_pos={0 0}, upload=420682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:44:17.147493952 (479h15m17.018576987s) [upload=420682]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:44:17.147498631 (479h15m17.018581666s) [upload=420682]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:44:17.435920485 (479h15m17.30700353s) [upload=420682]: Done uploading segments...
Dec 25 14:44:17.435940744 (479h15m17.307023779s) [upload=420682]: Upload finished. err: <nil>
Duration: 54.222121652s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................................XX....X...] duration:
7.390958928s trace:
https://jaeger.dev.storj.io/trace/6c4af9bed16efc90
FAIL (2025-12-25 14:48:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
egments...
Dec 25 14:46:13.264075569 (479h17m13.135158614s) [upload=420712]: Got next segment. Inline: false
Dec 25 14:46:13.264162195 (479h17m13.13524524s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:46:13.264170851 (479h17m13.135253886s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:46:13.512525890 (479h17m13.383608946s) [seg_pos={0 0}, upload=420712]: Begin upload segment...
Dec 25 14:46:13.512581887 (479h17m13.383664943s) [seg_pos={0 0}, upload=420712]: Done begin upload segment.
Dec 25 14:46:13.512618988 (479h17m13.383702063s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.512636512 (479h17m13.383719547s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.513623511 (479h17m13.384706556s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514215328 (479h17m13.385298373s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514707735 (479h17m13.38579078s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515215002 (479h17m13.386298047s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515714012 (479h17m13.386797057s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516240404 (479h17m13.387323449s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516782386 (479h17m13.387865441s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517289491 (479h17m13.388372536s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517780436 (479h17m13.388863501s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517871841 (479h17m13.388954916s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518404334 (479h17m13.389487379s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518876013 (479h17m13.389959058s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519390463 (479h17m13.390473508s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519862140 (479h17m13.390945185s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:15.229763881 (479h17m15.100846916s) [upload=420712]: Next returned nil segment
Dec 25 14:46:15.229870384 (479h17m15.100953429s) [upload=420712]: Waiting for error group managing segments...
Dec 25 14:46:15.513913870 (479h17m15.384996905s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.525529769 (479h17m15.396612814s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.535294673 (479h17m15.406377718s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536111188 (479h17m15.407194233s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536510157 (479h17m15.407593203s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.537886388 (479h17m15.408969434s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.551463891 (479h17m15.422546936s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.557990359 (479h17m15.429073404s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.558774191 (479h17m15.429857226s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.609636352 (479h17m15.480719397s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.747129702 (479h17m15.618212747s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.800198454 (479h17m15.671281499s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853951790 (479h17m15.725034825s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853973741 (479h17m15.725056776s) [seg_pos={0 0}, upload=420712]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:46:15.854155878 (479h17m15.725238923s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection
Dec 25 14:46:15.854164465 (479h17m15.72524751s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854293810 (479h17m15.725376856s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 25 14:46:15.854306345 (479h17m15.72538938s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854356590 (479h17m15.725439625s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection
Dec 25 14:46:15.854363533 (479h17m15.725446568s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854377661 (479h17m15.725460706s) [seg_pos={0 0}, upload=420712]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:46:15.854448205 (479h17m15.72553134s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:46:15.854452914 (479h17m15.725535959s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:46:16.148710570 (479h17m16.019793605s) [upload=420712]: Done uploading segments...
Dec 25 14:46:16.148734706 (479h17m16.019817752s) [upload=420712]: Upload finished. err: <nil>
Duration: 7.390988043s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
18 out of 60 is failed (threshold=6) [.XXXXXX.......X....X....X......XX......X......X..X.....XXXX.] duration:
3.521391763s trace:
https://jaeger.dev.storj.io/trace/63639b1450461448
FAIL (2025-12-25 14:48:21 UTC)
orbiter warning: context deadline exceeded
Duration: 3.521421579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................................XX....X..] duration:
5.916018044s trace:
https://jaeger.dev.storj.io/trace/6c4af9bed16efc90
FAIL (2025-12-25 14:48:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
egments...
Dec 25 14:46:13.264075569 (479h17m13.135158614s) [upload=420712]: Got next segment. Inline: false
Dec 25 14:46:13.264162195 (479h17m13.13524524s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:46:13.264170851 (479h17m13.135253886s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:46:13.512525890 (479h17m13.383608946s) [seg_pos={0 0}, upload=420712]: Begin upload segment...
Dec 25 14:46:13.512581887 (479h17m13.383664943s) [seg_pos={0 0}, upload=420712]: Done begin upload segment.
Dec 25 14:46:13.512618988 (479h17m13.383702063s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.512636512 (479h17m13.383719547s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.513623511 (479h17m13.384706556s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514215328 (479h17m13.385298373s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514707735 (479h17m13.38579078s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515215002 (479h17m13.386298047s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515714012 (479h17m13.386797057s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516240404 (479h17m13.387323449s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516782386 (479h17m13.387865441s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517289491 (479h17m13.388372536s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517780436 (479h17m13.388863501s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517871841 (479h17m13.388954916s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518404334 (479h17m13.389487379s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518876013 (479h17m13.389959058s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519390463 (479h17m13.390473508s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519862140 (479h17m13.390945185s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:15.229763881 (479h17m15.100846916s) [upload=420712]: Next returned nil segment
Dec 25 14:46:15.229870384 (479h17m15.100953429s) [upload=420712]: Waiting for error group managing segments...
Dec 25 14:46:15.513913870 (479h17m15.384996905s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.525529769 (479h17m15.396612814s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.535294673 (479h17m15.406377718s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536111188 (479h17m15.407194233s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536510157 (479h17m15.407593203s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.537886388 (479h17m15.408969434s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.551463891 (479h17m15.422546936s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.557990359 (479h17m15.429073404s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.558774191 (479h17m15.429857226s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.609636352 (479h17m15.480719397s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.747129702 (479h17m15.618212747s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.800198454 (479h17m15.671281499s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853951790 (479h17m15.725034825s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853973741 (479h17m15.725056776s) [seg_pos={0 0}, upload=420712]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:46:15.854155878 (479h17m15.725238923s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection
Dec 25 14:46:15.854164465 (479h17m15.72524751s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854293810 (479h17m15.725376856s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 25 14:46:15.854306345 (479h17m15.72538938s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854356590 (479h17m15.725439625s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection
Dec 25 14:46:15.854363533 (479h17m15.725446568s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854377661 (479h17m15.725460706s) [seg_pos={0 0}, upload=420712]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:46:15.854448205 (479h17m15.72553134s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:46:15.854452914 (479h17m15.725535959s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:46:16.148710570 (479h17m16.019793605s) [upload=420712]: Done uploading segments...
Dec 25 14:46:16.148734706 (479h17m16.019817752s) [upload=420712]: Upload finished. err: <nil>
Duration: 5.916057729s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
10.18568031s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:47:56 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.185703835s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
35.96591261s trace:
https://jaeger.dev.storj.io/trace/6fe4241072c742e3
FAIL (2025-12-25 14:47:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.48:4305 , piece_id=OG4GALOR3LJX2D224H7ZU7ZNMWFZA2DOYUDK75PK7RIJJZNC3ORQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292399703 (479h15m14.163482759s) [noise=true , address=64.49.13.140:4510 , piece_id=QMZV6CYJ3XFRXGD7TXNUNW56MZMVC52QTVEQAP7FNGD5GYBSVL3A, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292475207 (479h15m14.163558242s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292983274 (479h15m14.164066319s) [noise=true , address=145.79.227.35:4515 , piece_id=234QBCMNJQNPWR5SCI37NIMWD5XDUUYGKVNG6LA2DFOEV53WZZXA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293076572 (479h15m14.164159617s) [noise=true , address=145.79.227.30:4776 , piece_id=3BXXLR3PGZE53TQBQMHJT4O7UVFGCZNQALAFPCGEW3KLYSI2PMFA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293469279 (479h15m14.164552324s) [noise=true , address=45.91.116.6:4003 , piece_id=WFKJKH56B6JB2IAQRZJAEXOJAUYFD2YTFKNSZM6UYWCYOXMO72YA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293653110 (479h15m14.164736155s) [noise=true , address=145.79.227.3:4178 , piece_id=ISPXIDTSKSSG2R6CXR6YSXKUEGOCMG4ADIBXZZM6PJMPNLHMIWLA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294018936 (479h15m14.165101971s) [noise=true , address=77.90.31.17:4801 , piece_id=ABQWE7YGCVPYL6WD3TQQ7IYCLFUBTHK3G3LNQRHJRCDQN2PNXBAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294278030 (479h15m14.165361075s) [noise=true , address=45.91.116.12:4241 , piece_id=SXQDJRNWL3MNKKSMGR2SALBGUT5ES7EGD5HIY2NSGATTFBZTDLAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294534178 (479h15m14.165617223s) [noise=true , address=145.79.227.44:4225 , piece_id=LSBRYB2DMJCHZKBSGBP26FNM55VKRBZXFRCYC77CSYTU2JZ7PT7Q, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294981219 (479h15m14.166064264s) [noise=true , address=77.90.31.15:4723 , piece_id=NKKKWYITQV2SEZNZHPZFVY4DU4G24ZGPYUTWPTRX7U7K27GSGBAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.295163165 (479h15m14.16624621s) [noise=true , address=77.90.31.18:4836 , piece_id=7XYLKDNPRABN3BS3V375CXCT4NEWUW7RF3NRVVPTO4YAJFFOIPQQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.295595989 (479h15m14.166679034s) [noise=true , address=77.90.31.12:4610 , piece_id=6CK7RGQKSVTL5PB6Q2NRWCCCRYNMRFDUWXNWWRHXJ4WH523Y55DQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.296289399 (479h15m14.167372444s) [noise=true , address=45.91.116.11:4123 , piece_id=NKCCXXNYBKFWIREAP7GUQKYQQHFF7M5NDXO3XUSHYR2S4QJVBUNA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.296903900 (479h15m14.167986945s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.297479797 (479h15m14.168562842s) [noise=true , address=77.90.31.13:4654 , piece_id=WQ5KVXNI73SQWFXBQWTZ2JQFN2Y7KNYIUZZEI5RCFW3YLKQLAQGA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:15.616863706 (479h15m15.487946751s) [upload=420682]: Next returned nil segment
Dec 25 14:44:15.616879916 (479h15m15.487962951s) [upload=420682]: Waiting for error group managing segments...
Dec 25 14:44:15.950058414 (479h15m15.821141459s) [noise=true , address=64.49.13.140:4510 , piece_id=QMZV6CYJ3XFRXGD7TXNUNW56MZMVC52QTVEQAP7FNGD5GYBSVL3A, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.187623839 (479h15m16.058706884s) [noise=true , address=145.79.227.48:4305 , piece_id=OG4GALOR3LJX2D224H7ZU7ZNMWFZA2DOYUDK75PK7RIJJZNC3ORQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.195138668 (479h15m16.066221703s) [noise=true , address=145.79.227.3:4178 , piece_id=ISPXIDTSKSSG2R6CXR6YSXKUEGOCMG4ADIBXZZM6PJMPNLHMIWLA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.350781588 (479h15m16.221864623s) [noise=true , address=77.90.31.12:4610 , piece_id=6CK7RGQKSVTL5PB6Q2NRWCCCRYNMRFDUWXNWWRHXJ4WH523Y55DQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.396653487 (479h15m16.267736522s) [noise=true , address=45.91.116.11:4123 , piece_id=NKCCXXNYBKFWIREAP7GUQKYQQHFF7M5NDXO3XUSHYR2S4QJVBUNA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.431748113 (479h15m16.302831158s) [noise=true , address=45.91.116.6:4003 , piece_id=WFKJKH56B6JB2IAQRZJAEXOJAUYFD2YTFKNSZM6UYWCYOXMO72YA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.432579556 (479h15m16.303662591s) [noise=true , address=77.90.31.13:4654 , piece_id=WQ5KVXNI73SQWFXBQWTZ2JQFN2Y7KNYIUZZEI5RCFW3YLKQLAQGA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.463009770 (479h15m16.334092815s) [noise=true , address=77.90.31.17:4801 , piece_id=ABQWE7YGCVPYL6WD3TQQ7IYCLFUBTHK3G3LNQRHJRCDQN2PNXBAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.560546326 (479h15m16.431629361s) [noise=true , address=145.79.227.35:4515 , piece_id=234QBCMNJQNPWR5SCI37NIMWD5XDUUYGKVNG6LA2DFOEV53WZZXA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.611247740 (479h15m16.482330786s) [noise=true , address=77.90.31.15:4723 , piece_id=NKKKWYITQV2SEZNZHPZFVY4DU4G24ZGPYUTWPTRX7U7K27GSGBAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.758818880 (479h15m16.629901925s) [noise=true , address=77.90.31.18:4836 , piece_id=7XYLKDNPRABN3BS3V375CXCT4NEWUW7RF3NRVVPTO4YAJFFOIPQQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.782424619 (479h15m16.653507665s) [noise=true , address=45.91.116.12:4241 , piece_id=SXQDJRNWL3MNKKSMGR2SALBGUT5ES7EGD5HIY2NSGATTFBZTDLAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.072910743 (479h15m16.943993789s) [noise=true , address=145.79.227.30:4776 , piece_id=3BXXLR3PGZE53TQBQMHJT4O7UVFGCZNQALAFPCGEW3KLYSI2PMFA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.146569922 (479h15m17.017652967s) [noise=true , address=145.79.227.44:4225 , piece_id=LSBRYB2DMJCHZKBSGBP26FNM55VKRBZXFRCYC77CSYTU2JZ7PT7Q, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.146599148 (479h15m17.017682183s) [seg_pos={0 0}, upload=420682]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:44:17.146770013 (479h15m17.017853059s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.146776546 (479h15m17.017859581s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147100592 (479h15m17.018183638s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection
Dec 25 14:44:17.147108839 (479h15m17.018191884s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147251260 (479h15m17.018334305s) [noise=true , address=145.79.227.34:4915 , piece_id=GR66BPUQOFREGS2ZE36OB34SWX22AXIUASSDPIHR45ITW2F7ODCA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.147260467 (479h15m17.018343512s) [noise=true , address=145.79.227.34:4915 , piece_id=GR66BPUQOFREGS2ZE36OB34SWX22AXIUASSDPIHR45ITW2F7ODCA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147397388 (479h15m17.018480423s) [noise=true , address=77.90.31.11:4574 , piece_id=IJPN6SEXZWUEDR5DNBKP4RG3CJSXCZ7EEBSUSVQAGIEA54PAAPRA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.147405924 (479h15m17.018488969s) [noise=true , address=77.90.31.11:4574 , piece_id=IJPN6SEXZWUEDR5DNBKP4RG3CJSXCZ7EEBSUSVQAGIEA54PAAPRA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147425211 (479h15m17.018508256s) [seg_pos={0 0}, upload=420682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:44:17.147493952 (479h15m17.018576987s) [upload=420682]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:44:17.147498631 (479h15m17.018581666s) [upload=420682]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:44:17.435920485 (479h15m17.30700353s) [upload=420682]: Done uploading segments...
Dec 25 14:44:17.435940744 (479h15m17.307023779s) [upload=420682]: Upload finished. err: <nil>
Duration: 35.965928141s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
18 out of 60 is failed (threshold=6) [..XXXXXX.......X....X....X......XX......X......X..X.....XXXX] duration:
5.000817631s trace:
https://jaeger.dev.storj.io/trace/63639b1450461448
FAIL (2025-12-25 14:47:18 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000911979s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................................XX....X.] duration:
5.742075329s trace:
https://jaeger.dev.storj.io/trace/6c4af9bed16efc90
FAIL (2025-12-25 14:47:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
egments...
Dec 25 14:46:13.264075569 (479h17m13.135158614s) [upload=420712]: Got next segment. Inline: false
Dec 25 14:46:13.264162195 (479h17m13.13524524s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:46:13.264170851 (479h17m13.135253886s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:46:13.512525890 (479h17m13.383608946s) [seg_pos={0 0}, upload=420712]: Begin upload segment...
Dec 25 14:46:13.512581887 (479h17m13.383664943s) [seg_pos={0 0}, upload=420712]: Done begin upload segment.
Dec 25 14:46:13.512618988 (479h17m13.383702063s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.512636512 (479h17m13.383719547s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.513623511 (479h17m13.384706556s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514215328 (479h17m13.385298373s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514707735 (479h17m13.38579078s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515215002 (479h17m13.386298047s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515714012 (479h17m13.386797057s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516240404 (479h17m13.387323449s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516782386 (479h17m13.387865441s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517289491 (479h17m13.388372536s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517780436 (479h17m13.388863501s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517871841 (479h17m13.388954916s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518404334 (479h17m13.389487379s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518876013 (479h17m13.389959058s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519390463 (479h17m13.390473508s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519862140 (479h17m13.390945185s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:15.229763881 (479h17m15.100846916s) [upload=420712]: Next returned nil segment
Dec 25 14:46:15.229870384 (479h17m15.100953429s) [upload=420712]: Waiting for error group managing segments...
Dec 25 14:46:15.513913870 (479h17m15.384996905s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.525529769 (479h17m15.396612814s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.535294673 (479h17m15.406377718s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536111188 (479h17m15.407194233s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536510157 (479h17m15.407593203s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.537886388 (479h17m15.408969434s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.551463891 (479h17m15.422546936s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.557990359 (479h17m15.429073404s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.558774191 (479h17m15.429857226s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.609636352 (479h17m15.480719397s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.747129702 (479h17m15.618212747s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.800198454 (479h17m15.671281499s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853951790 (479h17m15.725034825s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853973741 (479h17m15.725056776s) [seg_pos={0 0}, upload=420712]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:46:15.854155878 (479h17m15.725238923s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection
Dec 25 14:46:15.854164465 (479h17m15.72524751s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854293810 (479h17m15.725376856s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 25 14:46:15.854306345 (479h17m15.72538938s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854356590 (479h17m15.725439625s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection
Dec 25 14:46:15.854363533 (479h17m15.725446568s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854377661 (479h17m15.725460706s) [seg_pos={0 0}, upload=420712]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:46:15.854448205 (479h17m15.72553134s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:46:15.854452914 (479h17m15.725535959s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:46:16.148710570 (479h17m16.019793605s) [upload=420712]: Done uploading segments...
Dec 25 14:46:16.148734706 (479h17m16.019817752s) [upload=420712]: Upload finished. err: <nil>
Duration: 5.742104875s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
12.653013536s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:47:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.653036851s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [...XXXXXX.......X....X....X......XX......X......X..X.....XXX] duration:
5.010668156s trace:
https://jaeger.dev.storj.io/trace/7f39619d32f39705
FAIL (2025-12-25 14:46:57 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.010714033s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
45.284864688s trace:
https://jaeger.dev.storj.io/trace/6fe4241072c742e3
FAIL (2025-12-25 14:46:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.48:4305 , piece_id=OG4GALOR3LJX2D224H7ZU7ZNMWFZA2DOYUDK75PK7RIJJZNC3ORQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292399703 (479h15m14.163482759s) [noise=true , address=64.49.13.140:4510 , piece_id=QMZV6CYJ3XFRXGD7TXNUNW56MZMVC52QTVEQAP7FNGD5GYBSVL3A, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292475207 (479h15m14.163558242s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292983274 (479h15m14.164066319s) [noise=true , address=145.79.227.35:4515 , piece_id=234QBCMNJQNPWR5SCI37NIMWD5XDUUYGKVNG6LA2DFOEV53WZZXA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293076572 (479h15m14.164159617s) [noise=true , address=145.79.227.30:4776 , piece_id=3BXXLR3PGZE53TQBQMHJT4O7UVFGCZNQALAFPCGEW3KLYSI2PMFA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293469279 (479h15m14.164552324s) [noise=true , address=45.91.116.6:4003 , piece_id=WFKJKH56B6JB2IAQRZJAEXOJAUYFD2YTFKNSZM6UYWCYOXMO72YA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293653110 (479h15m14.164736155s) [noise=true , address=145.79.227.3:4178 , piece_id=ISPXIDTSKSSG2R6CXR6YSXKUEGOCMG4ADIBXZZM6PJMPNLHMIWLA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294018936 (479h15m14.165101971s) [noise=true , address=77.90.31.17:4801 , piece_id=ABQWE7YGCVPYL6WD3TQQ7IYCLFUBTHK3G3LNQRHJRCDQN2PNXBAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294278030 (479h15m14.165361075s) [noise=true , address=45.91.116.12:4241 , piece_id=SXQDJRNWL3MNKKSMGR2SALBGUT5ES7EGD5HIY2NSGATTFBZTDLAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294534178 (479h15m14.165617223s) [noise=true , address=145.79.227.44:4225 , piece_id=LSBRYB2DMJCHZKBSGBP26FNM55VKRBZXFRCYC77CSYTU2JZ7PT7Q, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294981219 (479h15m14.166064264s) [noise=true , address=77.90.31.15:4723 , piece_id=NKKKWYITQV2SEZNZHPZFVY4DU4G24ZGPYUTWPTRX7U7K27GSGBAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.295163165 (479h15m14.16624621s) [noise=true , address=77.90.31.18:4836 , piece_id=7XYLKDNPRABN3BS3V375CXCT4NEWUW7RF3NRVVPTO4YAJFFOIPQQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.295595989 (479h15m14.166679034s) [noise=true , address=77.90.31.12:4610 , piece_id=6CK7RGQKSVTL5PB6Q2NRWCCCRYNMRFDUWXNWWRHXJ4WH523Y55DQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.296289399 (479h15m14.167372444s) [noise=true , address=45.91.116.11:4123 , piece_id=NKCCXXNYBKFWIREAP7GUQKYQQHFF7M5NDXO3XUSHYR2S4QJVBUNA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.296903900 (479h15m14.167986945s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.297479797 (479h15m14.168562842s) [noise=true , address=77.90.31.13:4654 , piece_id=WQ5KVXNI73SQWFXBQWTZ2JQFN2Y7KNYIUZZEI5RCFW3YLKQLAQGA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:15.616863706 (479h15m15.487946751s) [upload=420682]: Next returned nil segment
Dec 25 14:44:15.616879916 (479h15m15.487962951s) [upload=420682]: Waiting for error group managing segments...
Dec 25 14:44:15.950058414 (479h15m15.821141459s) [noise=true , address=64.49.13.140:4510 , piece_id=QMZV6CYJ3XFRXGD7TXNUNW56MZMVC52QTVEQAP7FNGD5GYBSVL3A, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.187623839 (479h15m16.058706884s) [noise=true , address=145.79.227.48:4305 , piece_id=OG4GALOR3LJX2D224H7ZU7ZNMWFZA2DOYUDK75PK7RIJJZNC3ORQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.195138668 (479h15m16.066221703s) [noise=true , address=145.79.227.3:4178 , piece_id=ISPXIDTSKSSG2R6CXR6YSXKUEGOCMG4ADIBXZZM6PJMPNLHMIWLA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.350781588 (479h15m16.221864623s) [noise=true , address=77.90.31.12:4610 , piece_id=6CK7RGQKSVTL5PB6Q2NRWCCCRYNMRFDUWXNWWRHXJ4WH523Y55DQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.396653487 (479h15m16.267736522s) [noise=true , address=45.91.116.11:4123 , piece_id=NKCCXXNYBKFWIREAP7GUQKYQQHFF7M5NDXO3XUSHYR2S4QJVBUNA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.431748113 (479h15m16.302831158s) [noise=true , address=45.91.116.6:4003 , piece_id=WFKJKH56B6JB2IAQRZJAEXOJAUYFD2YTFKNSZM6UYWCYOXMO72YA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.432579556 (479h15m16.303662591s) [noise=true , address=77.90.31.13:4654 , piece_id=WQ5KVXNI73SQWFXBQWTZ2JQFN2Y7KNYIUZZEI5RCFW3YLKQLAQGA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.463009770 (479h15m16.334092815s) [noise=true , address=77.90.31.17:4801 , piece_id=ABQWE7YGCVPYL6WD3TQQ7IYCLFUBTHK3G3LNQRHJRCDQN2PNXBAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.560546326 (479h15m16.431629361s) [noise=true , address=145.79.227.35:4515 , piece_id=234QBCMNJQNPWR5SCI37NIMWD5XDUUYGKVNG6LA2DFOEV53WZZXA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.611247740 (479h15m16.482330786s) [noise=true , address=77.90.31.15:4723 , piece_id=NKKKWYITQV2SEZNZHPZFVY4DU4G24ZGPYUTWPTRX7U7K27GSGBAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.758818880 (479h15m16.629901925s) [noise=true , address=77.90.31.18:4836 , piece_id=7XYLKDNPRABN3BS3V375CXCT4NEWUW7RF3NRVVPTO4YAJFFOIPQQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.782424619 (479h15m16.653507665s) [noise=true , address=45.91.116.12:4241 , piece_id=SXQDJRNWL3MNKKSMGR2SALBGUT5ES7EGD5HIY2NSGATTFBZTDLAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.072910743 (479h15m16.943993789s) [noise=true , address=145.79.227.30:4776 , piece_id=3BXXLR3PGZE53TQBQMHJT4O7UVFGCZNQALAFPCGEW3KLYSI2PMFA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.146569922 (479h15m17.017652967s) [noise=true , address=145.79.227.44:4225 , piece_id=LSBRYB2DMJCHZKBSGBP26FNM55VKRBZXFRCYC77CSYTU2JZ7PT7Q, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.146599148 (479h15m17.017682183s) [seg_pos={0 0}, upload=420682]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:44:17.146770013 (479h15m17.017853059s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.146776546 (479h15m17.017859581s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147100592 (479h15m17.018183638s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection
Dec 25 14:44:17.147108839 (479h15m17.018191884s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147251260 (479h15m17.018334305s) [noise=true , address=145.79.227.34:4915 , piece_id=GR66BPUQOFREGS2ZE36OB34SWX22AXIUASSDPIHR45ITW2F7ODCA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.147260467 (479h15m17.018343512s) [noise=true , address=145.79.227.34:4915 , piece_id=GR66BPUQOFREGS2ZE36OB34SWX22AXIUASSDPIHR45ITW2F7ODCA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147397388 (479h15m17.018480423s) [noise=true , address=77.90.31.11:4574 , piece_id=IJPN6SEXZWUEDR5DNBKP4RG3CJSXCZ7EEBSUSVQAGIEA54PAAPRA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.147405924 (479h15m17.018488969s) [noise=true , address=77.90.31.11:4574 , piece_id=IJPN6SEXZWUEDR5DNBKP4RG3CJSXCZ7EEBSUSVQAGIEA54PAAPRA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147425211 (479h15m17.018508256s) [seg_pos={0 0}, upload=420682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:44:17.147493952 (479h15m17.018576987s) [upload=420682]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:44:17.147498631 (479h15m17.018581666s) [upload=420682]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:44:17.435920485 (479h15m17.30700353s) [upload=420682]: Done uploading segments...
Dec 25 14:44:17.435940744 (479h15m17.307023779s) [upload=420682]: Upload finished. err: <nil>
Duration: 45.284885339s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................XX....X] duration:
30.00329403s trace:
https://jaeger.dev.storj.io/trace/6c4af9bed16efc90
FAIL (2025-12-25 14:46:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
egments...
Dec 25 14:46:13.264075569 (479h17m13.135158614s) [upload=420712]: Got next segment. Inline: false
Dec 25 14:46:13.264162195 (479h17m13.13524524s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:46:13.264170851 (479h17m13.135253886s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:46:13.512525890 (479h17m13.383608946s) [seg_pos={0 0}, upload=420712]: Begin upload segment...
Dec 25 14:46:13.512581887 (479h17m13.383664943s) [seg_pos={0 0}, upload=420712]: Done begin upload segment.
Dec 25 14:46:13.512618988 (479h17m13.383702063s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.512636512 (479h17m13.383719547s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.513623511 (479h17m13.384706556s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514215328 (479h17m13.385298373s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.514707735 (479h17m13.38579078s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515215002 (479h17m13.386298047s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.515714012 (479h17m13.386797057s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516240404 (479h17m13.387323449s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.516782386 (479h17m13.387865441s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517289491 (479h17m13.388372536s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517780436 (479h17m13.388863501s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.517871841 (479h17m13.388954916s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518404334 (479h17m13.389487379s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.518876013 (479h17m13.389959058s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519390463 (479h17m13.390473508s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:13.519862140 (479h17m13.390945185s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Uploading piece...
Dec 25 14:46:15.229763881 (479h17m15.100846916s) [upload=420712]: Next returned nil segment
Dec 25 14:46:15.229870384 (479h17m15.100953429s) [upload=420712]: Waiting for error group managing segments...
Dec 25 14:46:15.513913870 (479h17m15.384996905s) [noise=true , address=145.79.227.5:4221 , piece_id=HUXDXK6UANRU2YV3NGXJJLJ2TYCPVEHYJOKTP74UO42EAZ5JBYPQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.525529769 (479h17m15.396612814s) [noise=true , address=145.79.227.16:4910 , piece_id=RG7THLO6BRAV6BTPE43HG5HOI5FKY4AU7BKNGZ7KSI66RPOPSH4Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.535294673 (479h17m15.406377718s) [noise=true , address=145.79.227.37:4566 , piece_id=S6ZOYSUPHHVC73F6ZO5ZNVIOSBBO3HHAJ4AOGQRUV4CIU57XN3KA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536111188 (479h17m15.407194233s) [noise=true , address=145.79.227.20:4437 , piece_id=PK2YSKCSNZ7TUIKF4XYLV3V45T2T3VWTU7FFVB3UNHWB6GIMKWOA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.536510157 (479h17m15.407593203s) [noise=true , address=145.79.227.15:4875 , piece_id=OTZ5PVNQVR5SYICLQOFDOWC3425FWIEOPMHOQUKR2QMPI24ISMQQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.537886388 (479h17m15.408969434s) [noise=true , address=145.79.227.12:4765 , piece_id=6Y6X3IEGNZZ5BGXBJEXS5GU3RBVLMNUFWQVLTWYXC2CVCTJSO6VQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.551463891 (479h17m15.422546936s) [noise=true , address=145.79.227.49:4339 , piece_id=QZ3RNDQGQQQLWMM43SROGCOBDAWNDL2R3SRC3PI5DYFHUEMCO7RA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.557990359 (479h17m15.429073404s) [noise=true , address=145.79.227.51:4411 , piece_id=76LTWITTR6XJIHMXQKN2RUGFLRPLSJBYKNAIVLFKXPYKJBBXERFA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.558774191 (479h17m15.429857226s) [noise=true , address=145.79.227.7:4293 , piece_id=XFCQCNQWOBVD4TMMTW5WH34ECFJGZGGLNPBOTFCVPTUXEXLAXVDQ, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.609636352 (479h17m15.480719397s) [noise=true , address=145.79.227.33:4870 , piece_id=EQIMWZNFH4WZC3RNGPCUTIFIR6UHBCTBEBMWBF5GIBPIBBAHOODA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.747129702 (479h17m15.618212747s) [noise=true , address=45.91.116.6:4975 , piece_id=32NPQYQZPVCWNGK2E3UWRB5YVHCTVJMEH6A3V4U5H6SKFQWPLAQA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.800198454 (479h17m15.671281499s) [noise=true , address=45.91.116.11:4117 , piece_id=IUDGZFXZ4A324PTOLCLDIJCDHKSCRDT63M7XPVVEZYPGLPTL336A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853951790 (479h17m15.725034825s) [noise=true , address=77.90.31.14:4662 , piece_id=TLLNB3G5PVWNHIRZ2B77LZPFSWUOFAULJ6KMMAPRY4VYAP2XV36Q, seg_pos={0 0}, upload=420712]: Done uploading piece. err: <nil>
Dec 25 14:46:15.853973741 (479h17m15.725056776s) [seg_pos={0 0}, upload=420712]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:46:15.854155878 (479h17m15.725238923s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55532->45.91.116.13:4153: use of closed network connection
Dec 25 14:46:15.854164465 (479h17m15.72524751s) [noise=true , address=45.91.116.13:4153 , piece_id=MHIKRUCNGLJFU723TLRYAUUOKULMHNYZDKNR3UGVK76VEDGRZ75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854293810 (479h17m15.725376856s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 25 14:46:15.854306345 (479h17m15.72538938s) [noise=true , address=45.91.116.9:4264 , piece_id=NTL245V4666NZSCODHT5WE2BNORMAMYK2RGJAT24IRY3YC7BCTAA, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854356590 (479h17m15.725439625s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->77.90.31.16:4736: use of closed network connection
Dec 25 14:46:15.854363533 (479h17m15.725446568s) [noise=true , address=77.90.31.16:4736 , piece_id=3XK7XHEOTLUY6AAW4Y5RYDZVKOVJ3E7LTIELV2DZDUOGA7O2J75A, seg_pos={0 0}, upload=420712]: Long tail cancellation due to optimal threshold
Dec 25 14:46:15.854377661 (479h17m15.725460706s) [seg_pos={0 0}, upload=420712]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:46:15.854448205 (479h17m15.72553134s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:46:15.854452914 (479h17m15.725535959s) [upload=420712]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:46:16.148710570 (479h17m16.019793605s) [upload=420712]: Done uploading segments...
Dec 25 14:46:16.148734706 (479h17m16.019817752s) [upload=420712]: Upload finished. err: <nil>
Duration: 30.003316693s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
11.181577486s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:46:29 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.181607845s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [X...XXXXXX.......X....X....X......XX......X......X..X.....XX] duration:
5.001539963s trace:
https://jaeger.dev.storj.io/trace/1196766f1bca48e9
FAIL (2025-12-25 14:46:21 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001565523s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
54.450072298s trace:
https://jaeger.dev.storj.io/trace/6fe4241072c742e3
FAIL (2025-12-25 14:46:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.48:4305 , piece_id=OG4GALOR3LJX2D224H7ZU7ZNMWFZA2DOYUDK75PK7RIJJZNC3ORQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292399703 (479h15m14.163482759s) [noise=true , address=64.49.13.140:4510 , piece_id=QMZV6CYJ3XFRXGD7TXNUNW56MZMVC52QTVEQAP7FNGD5GYBSVL3A, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292475207 (479h15m14.163558242s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.292983274 (479h15m14.164066319s) [noise=true , address=145.79.227.35:4515 , piece_id=234QBCMNJQNPWR5SCI37NIMWD5XDUUYGKVNG6LA2DFOEV53WZZXA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293076572 (479h15m14.164159617s) [noise=true , address=145.79.227.30:4776 , piece_id=3BXXLR3PGZE53TQBQMHJT4O7UVFGCZNQALAFPCGEW3KLYSI2PMFA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293469279 (479h15m14.164552324s) [noise=true , address=45.91.116.6:4003 , piece_id=WFKJKH56B6JB2IAQRZJAEXOJAUYFD2YTFKNSZM6UYWCYOXMO72YA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.293653110 (479h15m14.164736155s) [noise=true , address=145.79.227.3:4178 , piece_id=ISPXIDTSKSSG2R6CXR6YSXKUEGOCMG4ADIBXZZM6PJMPNLHMIWLA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294018936 (479h15m14.165101971s) [noise=true , address=77.90.31.17:4801 , piece_id=ABQWE7YGCVPYL6WD3TQQ7IYCLFUBTHK3G3LNQRHJRCDQN2PNXBAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294278030 (479h15m14.165361075s) [noise=true , address=45.91.116.12:4241 , piece_id=SXQDJRNWL3MNKKSMGR2SALBGUT5ES7EGD5HIY2NSGATTFBZTDLAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294534178 (479h15m14.165617223s) [noise=true , address=145.79.227.44:4225 , piece_id=LSBRYB2DMJCHZKBSGBP26FNM55VKRBZXFRCYC77CSYTU2JZ7PT7Q, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.294981219 (479h15m14.166064264s) [noise=true , address=77.90.31.15:4723 , piece_id=NKKKWYITQV2SEZNZHPZFVY4DU4G24ZGPYUTWPTRX7U7K27GSGBAA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.295163165 (479h15m14.16624621s) [noise=true , address=77.90.31.18:4836 , piece_id=7XYLKDNPRABN3BS3V375CXCT4NEWUW7RF3NRVVPTO4YAJFFOIPQQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.295595989 (479h15m14.166679034s) [noise=true , address=77.90.31.12:4610 , piece_id=6CK7RGQKSVTL5PB6Q2NRWCCCRYNMRFDUWXNWWRHXJ4WH523Y55DQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.296289399 (479h15m14.167372444s) [noise=true , address=45.91.116.11:4123 , piece_id=NKCCXXNYBKFWIREAP7GUQKYQQHFF7M5NDXO3XUSHYR2S4QJVBUNA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.296903900 (479h15m14.167986945s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:14.297479797 (479h15m14.168562842s) [noise=true , address=77.90.31.13:4654 , piece_id=WQ5KVXNI73SQWFXBQWTZ2JQFN2Y7KNYIUZZEI5RCFW3YLKQLAQGA, seg_pos={0 0}, upload=420682]: Uploading piece...
Dec 25 14:44:15.616863706 (479h15m15.487946751s) [upload=420682]: Next returned nil segment
Dec 25 14:44:15.616879916 (479h15m15.487962951s) [upload=420682]: Waiting for error group managing segments...
Dec 25 14:44:15.950058414 (479h15m15.821141459s) [noise=true , address=64.49.13.140:4510 , piece_id=QMZV6CYJ3XFRXGD7TXNUNW56MZMVC52QTVEQAP7FNGD5GYBSVL3A, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.187623839 (479h15m16.058706884s) [noise=true , address=145.79.227.48:4305 , piece_id=OG4GALOR3LJX2D224H7ZU7ZNMWFZA2DOYUDK75PK7RIJJZNC3ORQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.195138668 (479h15m16.066221703s) [noise=true , address=145.79.227.3:4178 , piece_id=ISPXIDTSKSSG2R6CXR6YSXKUEGOCMG4ADIBXZZM6PJMPNLHMIWLA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.350781588 (479h15m16.221864623s) [noise=true , address=77.90.31.12:4610 , piece_id=6CK7RGQKSVTL5PB6Q2NRWCCCRYNMRFDUWXNWWRHXJ4WH523Y55DQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.396653487 (479h15m16.267736522s) [noise=true , address=45.91.116.11:4123 , piece_id=NKCCXXNYBKFWIREAP7GUQKYQQHFF7M5NDXO3XUSHYR2S4QJVBUNA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.431748113 (479h15m16.302831158s) [noise=true , address=45.91.116.6:4003 , piece_id=WFKJKH56B6JB2IAQRZJAEXOJAUYFD2YTFKNSZM6UYWCYOXMO72YA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.432579556 (479h15m16.303662591s) [noise=true , address=77.90.31.13:4654 , piece_id=WQ5KVXNI73SQWFXBQWTZ2JQFN2Y7KNYIUZZEI5RCFW3YLKQLAQGA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.463009770 (479h15m16.334092815s) [noise=true , address=77.90.31.17:4801 , piece_id=ABQWE7YGCVPYL6WD3TQQ7IYCLFUBTHK3G3LNQRHJRCDQN2PNXBAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.560546326 (479h15m16.431629361s) [noise=true , address=145.79.227.35:4515 , piece_id=234QBCMNJQNPWR5SCI37NIMWD5XDUUYGKVNG6LA2DFOEV53WZZXA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.611247740 (479h15m16.482330786s) [noise=true , address=77.90.31.15:4723 , piece_id=NKKKWYITQV2SEZNZHPZFVY4DU4G24ZGPYUTWPTRX7U7K27GSGBAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.758818880 (479h15m16.629901925s) [noise=true , address=77.90.31.18:4836 , piece_id=7XYLKDNPRABN3BS3V375CXCT4NEWUW7RF3NRVVPTO4YAJFFOIPQQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:16.782424619 (479h15m16.653507665s) [noise=true , address=45.91.116.12:4241 , piece_id=SXQDJRNWL3MNKKSMGR2SALBGUT5ES7EGD5HIY2NSGATTFBZTDLAA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.072910743 (479h15m16.943993789s) [noise=true , address=145.79.227.30:4776 , piece_id=3BXXLR3PGZE53TQBQMHJT4O7UVFGCZNQALAFPCGEW3KLYSI2PMFA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.146569922 (479h15m17.017652967s) [noise=true , address=145.79.227.44:4225 , piece_id=LSBRYB2DMJCHZKBSGBP26FNM55VKRBZXFRCYC77CSYTU2JZ7PT7Q, seg_pos={0 0}, upload=420682]: Done uploading piece. err: <nil>
Dec 25 14:44:17.146599148 (479h15m17.017682183s) [seg_pos={0 0}, upload=420682]: Segment reached optimal threshold of 14 pieces.
Dec 25 14:44:17.146770013 (479h15m17.017853059s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.146776546 (479h15m17.017859581s) [noise=true , address=145.79.227.40:4062 , piece_id=SHL34MAC5PIELGW3EFJSBYFO5NQ45AJKXAN3AG7GWVO632MUHHSQ, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147100592 (479h15m17.018183638s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55104->64.49.13.169:4705: use of closed network connection
Dec 25 14:44:17.147108839 (479h15m17.018191884s) [noise=true , address=64.49.13.169:4705 , piece_id=EEEUZXSROJ6XVTKSWA2KM656WQC53MSBWEJGW6XZJ5I2HVDSACIA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147251260 (479h15m17.018334305s) [noise=true , address=145.79.227.34:4915 , piece_id=GR66BPUQOFREGS2ZE36OB34SWX22AXIUASSDPIHR45ITW2F7ODCA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.147260467 (479h15m17.018343512s) [noise=true , address=145.79.227.34:4915 , piece_id=GR66BPUQOFREGS2ZE36OB34SWX22AXIUASSDPIHR45ITW2F7ODCA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147397388 (479h15m17.018480423s) [noise=true , address=77.90.31.11:4574 , piece_id=IJPN6SEXZWUEDR5DNBKP4RG3CJSXCZ7EEBSUSVQAGIEA54PAAPRA, seg_pos={0 0}, upload=420682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF
Dec 25 14:44:17.147405924 (479h15m17.018488969s) [noise=true , address=77.90.31.11:4574 , piece_id=IJPN6SEXZWUEDR5DNBKP4RG3CJSXCZ7EEBSUSVQAGIEA54PAAPRA, seg_pos={0 0}, upload=420682]: Long tail cancellation due to optimal threshold
Dec 25 14:44:17.147425211 (479h15m17.018508256s) [seg_pos={0 0}, upload=420682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 25 14:44:17.147493952 (479h15m17.018576987s) [upload=420682]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:44:17.147498631 (479h15m17.018581666s) [upload=420682]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:44:17.435920485 (479h15m17.30700353s) [upload=420682]: Done uploading segments...
Dec 25 14:44:17.435940744 (479h15m17.307023779s) [upload=420682]: Upload finished. err: <nil>
Duration: 54.450087567s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [.X...XXXXXX.......X....X....X......XX......X......X..X.....X] duration:
5.000756142s trace:
https://jaeger.dev.storj.io/trace/71729c576d0d6818
FAIL (2025-12-25 14:45:48 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000777331s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
13.02538288s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:45:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.025408788s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................XX....] duration:
5.970169278s trace:
https://jaeger.dev.storj.io/trace/2464b09252149d1d
FAIL (2025-12-25 14:45:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q) context deadline exceeded
======LOGS======
Dec 25 14:41:53.719625840 (479h12m53.590708886s) [upload=420648]: Starting upload
Dec 25 14:41:53.719714038 (479h12m53.590797093s) [upload=420648]: Done starting upload
Dec 25 14:41:53.719815611 (479h12m53.590898656s) [upload=420648]: Uploading segments...
Dec 25 14:41:53.719987589 (479h12m53.591070624s) [upload=420648]: Got next segment. Inline: false
Dec 25 14:41:53.720028667 (479h12m53.591111702s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:41:53.720034468 (479h12m53.591117503s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:41:53.974566669 (479h12m53.845649724s) [seg_pos={0 0}, upload=420648]: Begin upload segment...
Dec 25 14:41:53.974622285 (479h12m53.845705331s) [seg_pos={0 0}, upload=420648]: Done begin upload segment.
Dec 25 14:41:53.974656471 (479h12m53.845739516s) [noise=true , address=145.79.227.50:4373 , piece_id=3K2NTODUNE6PYOHO45LOONWHTXCHFJUKT6YLHBG3QWTUG6CBKNRQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.974706936 (479h12m53.845789971s) [noise=true , address=145.79.227.14:4838 , piece_id=RJZBX327RFDUNC3QRAP5PZH3V4KOFZMZAQTXB56LZL7D4RQJTXEQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.975246945 (479h12m53.84632999s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBXKQSOBYUCXRDRTV3DF24CPC5NSU6GWM32JO2RQXCYLTCQN3WBA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.975622080 (479h12m53.846705125s) [noise=true , address=145.79.227.13:4802 , piece_id=ILEQKN33E7B3J2XKVFAUG5WGJEEB5ON63KL5IKYG63GSZV6BDOWA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976017071 (479h12m53.847100106s) [noise=true , address=145.79.227.3:4149 , piece_id=F44ZHK6VEBKCFMF7YB5UEMCQAXIDVCRHNBMIQHHZPTC7INY574RA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976412263 (479h12m53.847495329s) [noise=true , address=45.91.116.9:4263 , piece_id=SPH5AHYRGLHCOHYL56NGQNGMJFKJCI44CFO5HHWJKTHDIDGDRP4A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976803339 (479h12m53.847886384s) [noise=true , address=45.91.116.7:4010 , piece_id=DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.977290787 (479h12m53.848373822s) [noise=true , address=145.79.227.45:4231 , piece_id=YIESEOIYY4Q7S5ENJQWSRYRPVCV5FGQRDMBUILX6YK2OS3NXVS5A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.978431939 (479h12m53.849514984s) [noise=true , address=145.79.227.41:4087 , piece_id=ISXUYPLSDSF3XB6F5IKF3F2VCCWNAJXFLSDX5GXUMVONHB7DBV7Q, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.978544464 (479h12m53.849627509s) [noise=true , address=145.79.227.29:4943 , piece_id=R6ASHKZY6V3OERFGZ6ZAROQA4BG5VD5J3YLIVBNTDELI4SM7FVQA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.979619391 (479h12m53.850702426s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.980847840 (479h12m53.851930875s) [noise=true , address=145.79.227.52:4447 , piece_id=RHZAOUKYC4QAKICT2ZSEIGZIARL34CPRGV4NWG6LDRPOFXUTNRRA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.983019956 (479h12m53.854102991s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.984892983 (479h12m53.855976018s) [noise=true , address=45.91.116.12:4228 , piece_id=WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.986444197 (479h12m53.857527242s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.988382928 (479h12m53.859465973s) [noise=true , address=145.79.227.35:4482 , piece_id=ZF5MJ4G577NOYRI7KSIACJRMP3GPLMY7COAIUUO7HLRSBYUWJWOA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:55.656144797 (479h12m55.527227842s) [upload=420648]: Next returned nil segment
Dec 25 14:41:55.656159716 (479h12m55.527242761s) [upload=420648]: Waiting for error group managing segments...
Dec 25 14:41:55.980587370 (479h12m55.851670405s) [noise=true , address=145.79.227.13:4802 , piece_id=ILEQKN33E7B3J2XKVFAUG5WGJEEB5ON63KL5IKYG63GSZV6BDOWA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:55.980623830 (479h12m55.851706875s) [noise=true , address=145.79.227.3:4149 , piece_id=F44ZHK6VEBKCFMF7YB5UEMCQAXIDVCRHNBMIQHHZPTC7INY574RA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:55.994117394 (479h12m55.865200439s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBXKQSOBYUCXRDRTV3DF24CPC5NSU6GWM32JO2RQXCYLTCQN3WBA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.001610682 (479h12m55.872693738s) [noise=true , address=145.79.227.45:4231 , piece_id=YIESEOIYY4Q7S5ENJQWSRYRPVCV5FGQRDMBUILX6YK2OS3NXVS5A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004085315 (479h12m55.87516837s) [noise=true , address=145.79.227.29:4943 , piece_id=R6ASHKZY6V3OERFGZ6ZAROQA4BG5VD5J3YLIVBNTDELI4SM7FVQA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004143506 (479h12m55.875226571s) [noise=true , address=145.79.227.52:4447 , piece_id=RHZAOUKYC4QAKICT2ZSEIGZIARL34CPRGV4NWG6LDRPOFXUTNRRA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004725414 (479h12m55.875808469s) [noise=true , address=145.79.227.50:4373 , piece_id=3K2NTODUNE6PYOHO45LOONWHTXCHFJUKT6YLHBG3QWTUG6CBKNRQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.005029713 (479h12m55.876112778s) [noise=true , address=145.79.227.41:4087 , piece_id=ISXUYPLSDSF3XB6F5IKF3F2VCCWNAJXFLSDX5GXUMVONHB7DBV7Q, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.009615144 (479h12m55.88069819s) [noise=true , address=145.79.227.14:4838 , piece_id=RJZBX327RFDUNC3QRAP5PZH3V4KOFZMZAQTXB56LZL7D4RQJTXEQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.018810834 (479h12m55.88989388s) [noise=true , address=145.79.227.35:4482 , piece_id=ZF5MJ4G577NOYRI7KSIACJRMP3GPLMY7COAIUUO7HLRSBYUWJWOA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.302829424 (479h12m56.173912459s) [noise=true , address=45.91.116.9:4263 , piece_id=SPH5AHYRGLHCOHYL56NGQNGMJFKJCI44CFO5HHWJKTHDIDGDRP4A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.335735653 (479h12m56.206818698s) [noise=true , address=45.91.116.7:4010 , piece_id=DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.668297637 (479h12m56.539380682s) [noise=true , address=45.91.116.12:4228 , piece_id=WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.668326472 (479h12m56.539409517s) [seg_pos={0 0}, upload=420648]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:41:56.668678312 (479h12m56.539761377s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.668689994 (479h12m56.539773039s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669182391 (479h12m56.540265436s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.669212619 (479h12m56.540295664s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669456283 (479h12m56.540539328s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.669487041 (479h12m56.540570077s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669512500 (479h12m56.540595535s) [seg_pos={0 0}, upload=420648]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:41:56.669628441 (479h12m56.540711486s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:41:56.669655252 (479h12m56.540738297s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:41:56.965564873 (479h12m56.836647928s) [upload=420648]: Done uploading segments...
Dec 25 14:41:56.965590703 (479h12m56.836673748s) [upload=420648]: Upload finished. err: <nil>
Duration: 5.970198542s
core
uplink
us-select-1
|